Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250623 08:04:18.003667 32020 test_util.cc:276] Using random seed: -1466551644
2025-06-23T08:04:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-23T08:04:18Z Disabled control of system clock
I20250623 08:04:18.037467 32020 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskJxinPB/build/asan/bin/kudu
/tmp/dist-test-taskJxinPB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.62:39465
--webserver_interface=127.31.69.62
--webserver_port=0
--builtin_ntp_servers=127.31.69.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250623 08:04:18.352614 32028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 08:04:18.352932 32028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 08:04:18.353019 32028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 08:04:18.360697 32028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 08:04:18.360790 32028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 08:04:18.360884 32028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 08:04:18.360952 32028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 08:04:18.370343 32028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.69.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.62:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.69.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 07:50:47 UTC on 24a791456cd2
build id 6728
ASAN enabled
I20250623 08:04:18.372105 32028 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 08:04:18.373900 32028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 08:04:18.384949 32034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:18.385012 32033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:18.384984 32036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:18.386436 32028 server_base.cc:1048] running on GCE node
I20250623 08:04:18.387223 32028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 08:04:18.388677 32028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 08:04:18.389881 32028 hybrid_clock.cc:648] HybridClock initialized: now 1750665858389829 us; error 45 us; skew 500 ppm
I20250623 08:04:18.390223 32028 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 08:04:18.395298 32028 webserver.cc:469] Webserver started at http://127.31.69.62:33855/ using document root <none> and password file <none>
I20250623 08:04:18.396080 32028 fs_manager.cc:362] Metadata directory not provided
I20250623 08:04:18.396176 32028 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 08:04:18.396524 32028 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 08:04:18.399017 32028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data/instance:
uuid: "d31835ec4ad74332a00998d431492466"
format_stamp: "Formatted at 2025-06-23 08:04:18 on dist-test-slave-bxhn"
I20250623 08:04:18.399806 32028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal/instance:
uuid: "d31835ec4ad74332a00998d431492466"
format_stamp: "Formatted at 2025-06-23 08:04:18 on dist-test-slave-bxhn"
I20250623 08:04:18.405113 32028 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250623 08:04:18.408609 32042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:18.409968 32028 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20250623 08:04:18.410130 32028 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data,/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
uuid: "d31835ec4ad74332a00998d431492466"
format_stamp: "Formatted at 2025-06-23 08:04:18 on dist-test-slave-bxhn"
I20250623 08:04:18.410331 32028 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 08:04:18.452018 32028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 08:04:18.452955 32028 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 08:04:18.453694 32028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 08:04:18.473543 32028 rpc_server.cc:307] RPC server started. Bound to: 127.31.69.62:39465
I20250623 08:04:18.473569 32093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.69.62:39465 every 8 connection(s)
I20250623 08:04:18.475482 32028 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data/info.pb
I20250623 08:04:18.480633 32094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:18.484061 32020 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskJxinPB/build/asan/bin/kudu as pid 32028
I20250623 08:04:18.484375 32020 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal/instance
I20250623 08:04:18.487725 32020 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskJxinPB/build/asan/bin/kudu
/tmp/dist-test-taskJxinPB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.61:33313
--webserver_interface=127.31.69.61
--webserver_port=0
--builtin_ntp_servers=127.31.69.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250623 08:04:18.488937 32094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } has no permanent_uuid. Determining permanent_uuid...
I20250623 08:04:18.505934 32094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } has no permanent_uuid. Determining permanent_uuid...
W20250623 08:04:18.507134 32045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111)
W20250623 08:04:18.508670 32094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.61:33313: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111)
I20250623 08:04:18.558019 32094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } attempt: 1
W20250623 08:04:18.560145 32094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.61:33313: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111)
I20250623 08:04:18.628382 32094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } attempt: 2
W20250623 08:04:18.630124 32094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.61:33313: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111)
I20250623 08:04:18.721386 32094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } attempt: 3
W20250623 08:04:18.723167 32094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.61:33313: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111)
W20250623 08:04:18.804018 32096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 08:04:18.804353 32096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 08:04:18.804481 32096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 08:04:18.812096 32096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 08:04:18.812211 32096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 08:04:18.812325 32096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 08:04:18.812393 32096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 08:04:18.821520 32096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.69.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.61:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.69.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 07:50:47 UTC on 24a791456cd2
build id 6728
ASAN enabled
I20250623 08:04:18.823185 32096 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 08:04:18.825006 32096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 08:04:18.835146 32104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:18.835307 32105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:18.835619 32107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:18.836071 32096 server_base.cc:1048] running on GCE node
I20250623 08:04:18.836833 32096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 08:04:18.838275 32096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 08:04:18.839470 32096 hybrid_clock.cc:648] HybridClock initialized: now 1750665858839431 us; error 32 us; skew 500 ppm
I20250623 08:04:18.839805 32096 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 08:04:18.842108 32096 webserver.cc:469] Webserver started at http://127.31.69.61:44043/ using document root <none> and password file <none>
I20250623 08:04:18.842905 32096 fs_manager.cc:362] Metadata directory not provided
I20250623 08:04:18.843014 32096 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 08:04:18.843372 32096 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 08:04:18.845700 32096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data/instance:
uuid: "9752155c20ee42fbb7ff7b7d5b682444"
format_stamp: "Formatted at 2025-06-23 08:04:18 on dist-test-slave-bxhn"
I20250623 08:04:18.846437 32096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal/instance:
uuid: "9752155c20ee42fbb7ff7b7d5b682444"
format_stamp: "Formatted at 2025-06-23 08:04:18 on dist-test-slave-bxhn"
I20250623 08:04:18.851858 32096 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20250623 08:04:18.855175 32113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:18.857026 32096 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250623 08:04:18.857192 32096 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data,/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
uuid: "9752155c20ee42fbb7ff7b7d5b682444"
format_stamp: "Formatted at 2025-06-23 08:04:18 on dist-test-slave-bxhn"
I20250623 08:04:18.857399 32096 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 08:04:18.866385 32094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } attempt: 4
W20250623 08:04:18.868109 32094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.61:33313: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111)
I20250623 08:04:18.885397 32096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 08:04:18.886299 32096 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 08:04:18.886550 32096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 08:04:18.909659 32096 rpc_server.cc:307] RPC server started. Bound to: 127.31.69.61:33313
I20250623 08:04:18.909701 32164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.69.61:33313 every 8 connection(s)
I20250623 08:04:18.911140 32096 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data/info.pb
I20250623 08:04:18.915048 32165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:18.915527 32020 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskJxinPB/build/asan/bin/kudu as pid 32096
I20250623 08:04:18.915690 32020 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal/instance
I20250623 08:04:18.916772 32020 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskJxinPB/build/asan/bin/kudu
/tmp/dist-test-taskJxinPB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.60:39691
--webserver_interface=127.31.69.60
--webserver_port=0
--builtin_ntp_servers=127.31.69.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250623 08:04:18.920964 32165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } has no permanent_uuid. Determining permanent_uuid...
I20250623 08:04:18.934713 32165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } has no permanent_uuid. Determining permanent_uuid...
I20250623 08:04:18.943986 32165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
W20250623 08:04:18.944937 32114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
W20250623 08:04:18.946434 32165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:18.995694 32165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } attempt: 1
W20250623 08:04:18.997566 32165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:19.065800 32165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } attempt: 2
W20250623 08:04:19.067751 32165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:19.158983 32165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } attempt: 3
W20250623 08:04:19.160889 32165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:19.167361 32094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } attempt: 5
I20250623 08:04:19.174778 32094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
W20250623 08:04:19.176468 32094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:19.227700 32094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } attempt: 1
W20250623 08:04:19.229580 32094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
W20250623 08:04:19.245640 32167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 08:04:19.245983 32167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 08:04:19.246117 32167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 08:04:19.254529 32167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 08:04:19.254621 32167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 08:04:19.254699 32167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 08:04:19.254778 32167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 08:04:19.264451 32167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.69.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.60:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.69.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 07:50:47 UTC on 24a791456cd2
build id 6728
ASAN enabled
I20250623 08:04:19.266171 32167 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 08:04:19.268162 32167 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 08:04:19.278179 32175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:19.278231 32178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:19.278231 32176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:19.279194 32167 server_base.cc:1048] running on GCE node
I20250623 08:04:19.279974 32167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 08:04:19.281355 32167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 08:04:19.282546 32167 hybrid_clock.cc:648] HybridClock initialized: now 1750665859282479 us; error 64 us; skew 500 ppm
I20250623 08:04:19.282908 32167 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 08:04:19.285063 32167 webserver.cc:469] Webserver started at http://127.31.69.60:35145/ using document root <none> and password file <none>
I20250623 08:04:19.285794 32167 fs_manager.cc:362] Metadata directory not provided
I20250623 08:04:19.285884 32167 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 08:04:19.286226 32167 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 08:04:19.288597 32167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data/instance:
uuid: "6644af0f4b2a4b998282b021ec1dc678"
format_stamp: "Formatted at 2025-06-23 08:04:19 on dist-test-slave-bxhn"
I20250623 08:04:19.289364 32167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal/instance:
uuid: "6644af0f4b2a4b998282b021ec1dc678"
format_stamp: "Formatted at 2025-06-23 08:04:19 on dist-test-slave-bxhn"
I20250623 08:04:19.294790 32167 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250623 08:04:19.297819 32094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } attempt: 2
I20250623 08:04:19.298202 32184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250623 08:04:19.299613 32094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:19.299746 32167 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250623 08:04:19.299906 32167 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data,/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
uuid: "6644af0f4b2a4b998282b021ec1dc678"
format_stamp: "Formatted at 2025-06-23 08:04:19 on dist-test-slave-bxhn"
I20250623 08:04:19.300081 32167 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 08:04:19.304112 32165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } attempt: 4
W20250623 08:04:19.305642 32165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:19.329332 32167 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 08:04:19.330307 32167 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 08:04:19.330579 32167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 08:04:19.350805 32167 rpc_server.cc:307] RPC server started. Bound to: 127.31.69.60:39691
I20250623 08:04:19.350836 32235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.69.60:39691 every 8 connection(s)
I20250623 08:04:19.352383 32167 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data/info.pb
I20250623 08:04:19.353056 32020 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskJxinPB/build/asan/bin/kudu as pid 32167
I20250623 08:04:19.353262 32020 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal/instance
I20250623 08:04:19.356405 32236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:19.364334 32236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } has no permanent_uuid. Determining permanent_uuid...
I20250623 08:04:19.378774 32236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } has no permanent_uuid. Determining permanent_uuid...
I20250623 08:04:19.386073 32236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
I20250623 08:04:19.396827 32236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: Bootstrap starting.
I20250623 08:04:19.400616 32236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:19.401692 32236 log.cc:826] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: Log is configured to *not* fsync() on all Append() calls
I20250623 08:04:19.404258 32236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: No bootstrap required, opened a new log
I20250623 08:04:19.405819 32094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } attempt: 3
I20250623 08:04:19.410888 32236 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:19.411278 32236 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:19.411396 32236 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6644af0f4b2a4b998282b021ec1dc678, State: Initialized, Role: FOLLOWER
I20250623 08:04:19.412145 32236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:19.413096 32245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:19.413349 32245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:19.414435 32236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 08:04:19.417596 32094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Bootstrap starting.
I20250623 08:04:19.421980 32094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:19.423416 32094 log.cc:826] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Log is configured to *not* fsync() on all Append() calls
I20250623 08:04:19.425714 32094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: No bootstrap required, opened a new log
I20250623 08:04:19.422624 32167 master_runner.cc:186] Error getting master registration for 127.31.69.62:39465: OK, instance_id { permanent_uuid: "d31835ec4ad74332a00998d431492466" instance_seqno: 1750665858468179 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250623 08:04:19.430720 32167 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250623 08:04:19.431087 32094 raft_consensus.cc:357] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:19.431411 32094 raft_consensus.cc:383] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:19.431519 32094 raft_consensus.cc:738] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31835ec4ad74332a00998d431492466, State: Initialized, Role: FOLLOWER
I20250623 08:04:19.432242 32094 consensus_queue.cc:260] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:19.433147 32259 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
W20250623 08:04:19.433439 32256 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250623 08:04:19.433558 32256 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250623 08:04:19.433584 32259 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:19.434296 32094 sys_catalog.cc:564] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: configured and running, proceeding with master startup.
W20250623 08:04:19.444501 32270 catalog_manager.cc:1560] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250623 08:04:19.444634 32270 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250623 08:04:19.444028 32028 master_runner.cc:186] Error getting master registration for 127.31.69.61:33313: OK, instance_id { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" instance_seqno: 1750665858904185 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250623 08:04:19.450523 32028 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250623 08:04:19.456468 32048 tablet.cc:2378] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250623 08:04:19.582783 32190 tablet.cc:2378] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 08:04:19.604841 32165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } attempt: 5
I20250623 08:04:19.617713 32165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Bootstrap starting.
I20250623 08:04:19.621798 32165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:19.623179 32165 log.cc:826] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Log is configured to *not* fsync() on all Append() calls
I20250623 08:04:19.626142 32165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: No bootstrap required, opened a new log
I20250623 08:04:19.633317 32165 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:19.633663 32165 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:19.633764 32165 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9752155c20ee42fbb7ff7b7d5b682444, State: Initialized, Role: FOLLOWER
I20250623 08:04:19.635653 32165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:19.636778 32273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:19.637128 32273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:19.638121 32165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: configured and running, proceeding with master startup.
W20250623 08:04:19.641981 32119 tablet.cc:2378] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 08:04:19.660341 32096 master_runner.cc:263] No leader master found from master 9752155c20ee42fbb7ff7b7d5b682444
I20250623 08:04:19.660476 32096 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250623 08:04:19.663251 32284 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250623 08:04:19.663496 32284 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250623 08:04:19.769907 32245 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:19.770221 32245 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:19.772277 32245 leader_election.cc:290] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313)
I20250623 08:04:19.773736 32069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31835ec4ad74332a00998d431492466" is_pre_election: true
I20250623 08:04:19.774264 32069 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6644af0f4b2a4b998282b021ec1dc678 in term 0.
I20250623 08:04:19.775058 32185 leader_election.cc:304] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678, d31835ec4ad74332a00998d431492466; no voters:
I20250623 08:04:19.775592 32245 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:19.775728 32245 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:19.775570 32140 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
I20250623 08:04:19.775833 32245 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:19.776052 32140 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6644af0f4b2a4b998282b021ec1dc678 in term 0.
I20250623 08:04:19.782424 32245 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:19.784030 32069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31835ec4ad74332a00998d431492466"
I20250623 08:04:19.784243 32069 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:19.788293 32245 leader_election.cc:290] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 1 election: Requested vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313)
I20250623 08:04:19.788352 32140 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444"
I20250623 08:04:19.788566 32140 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:19.886936 32069 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6644af0f4b2a4b998282b021ec1dc678 in term 1.
I20250623 08:04:19.886933 32140 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6644af0f4b2a4b998282b021ec1dc678 in term 1.
I20250623 08:04:19.889204 32187 leader_election.cc:304] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678, 9752155c20ee42fbb7ff7b7d5b682444; no voters:
I20250623 08:04:19.892196 32245 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:19.899021 32245 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 1 LEADER]: Becoming Leader. State: Replica: 6644af0f4b2a4b998282b021ec1dc678, State: Running, Role: LEADER
I20250623 08:04:19.899685 32245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:19.906383 32286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6644af0f4b2a4b998282b021ec1dc678. Latest consensus state: current_term: 1 leader_uuid: "6644af0f4b2a4b998282b021ec1dc678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:19.906665 32286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: LEADER
I20250623 08:04:19.908062 32288 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 08:04:19.910238 32288 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 08:04:19.937192 32140 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Refusing update from remote peer 6644af0f4b2a4b998282b021ec1dc678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 08:04:19.937462 32069 raft_consensus.cc:1273] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Refusing update from remote peer 6644af0f4b2a4b998282b021ec1dc678: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 08:04:19.938158 32286 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:19.938810 32245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:19.956295 32273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6644af0f4b2a4b998282b021ec1dc678. Latest consensus state: current_term: 1 leader_uuid: "6644af0f4b2a4b998282b021ec1dc678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:19.956593 32273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:19.963557 32291 mvcc.cc:204] Tried to move back new op lower bound from 7170727362292633600 to 7170727362154323968. Current Snapshot: MvccSnapshot[applied={T|T < 7170727362292633600}]
I20250623 08:04:19.971031 32286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6644af0f4b2a4b998282b021ec1dc678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:19.971299 32286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: LEADER
I20250623 08:04:19.971755 32273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6644af0f4b2a4b998282b021ec1dc678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:19.971939 32273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:19.972091 32288 catalog_manager.cc:1349] Generated new cluster ID: bde910f56f35432e94ef19d006edb93c
I20250623 08:04:19.972189 32288 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 08:04:19.974653 32290 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6644af0f4b2a4b998282b021ec1dc678. Latest consensus state: current_term: 1 leader_uuid: "6644af0f4b2a4b998282b021ec1dc678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:19.974931 32290 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:19.975693 32286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6644af0f4b2a4b998282b021ec1dc678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:19.975988 32286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: LEADER
I20250623 08:04:19.992187 32290 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6644af0f4b2a4b998282b021ec1dc678" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:19.992431 32290 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:20.007551 32288 catalog_manager.cc:1372] Generated new certificate authority record
I20250623 08:04:20.008968 32288 catalog_manager.cc:1506] Loading token signing keys...
I20250623 08:04:20.019937 32288 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: Generated new TSK 0
I20250623 08:04:20.021087 32288 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 08:04:20.041153 32020 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskJxinPB/build/asan/bin/kudu
/tmp/dist-test-taskJxinPB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.69.1:0
--local_ip_for_outbound_sockets=127.31.69.1
--webserver_interface=127.31.69.1
--webserver_port=0
--tserver_master_addrs=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--builtin_ntp_servers=127.31.69.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250623 08:04:20.367733 32299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 08:04:20.368033 32299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 08:04:20.368124 32299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 08:04:20.375784 32299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 08:04:20.375950 32299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.69.1
I20250623 08:04:20.385058 32299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.69.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.69.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.69.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.69.1
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 07:50:47 UTC on 24a791456cd2
build id 6728
ASAN enabled
I20250623 08:04:20.386579 32299 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 08:04:20.388355 32299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 08:04:20.399852 32304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:20.399884 32305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:20.399936 32307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:20.401517 32299 server_base.cc:1048] running on GCE node
I20250623 08:04:20.402091 32299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 08:04:20.403141 32299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 08:04:20.404407 32299 hybrid_clock.cc:648] HybridClock initialized: now 1750665860404348 us; error 48 us; skew 500 ppm
I20250623 08:04:20.404719 32299 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 08:04:20.407080 32299 webserver.cc:469] Webserver started at http://127.31.69.1:35731/ using document root <none> and password file <none>
I20250623 08:04:20.407806 32299 fs_manager.cc:362] Metadata directory not provided
I20250623 08:04:20.407910 32299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 08:04:20.408241 32299 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 08:04:20.410557 32299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/data/instance:
uuid: "7351aa31800245659edc9837e27daffd"
format_stamp: "Formatted at 2025-06-23 08:04:20 on dist-test-slave-bxhn"
I20250623 08:04:20.411378 32299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/instance:
uuid: "7351aa31800245659edc9837e27daffd"
format_stamp: "Formatted at 2025-06-23 08:04:20 on dist-test-slave-bxhn"
I20250623 08:04:20.416468 32299 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20250623 08:04:20.419838 32313 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:20.421197 32299 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250623 08:04:20.421445 32299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal
uuid: "7351aa31800245659edc9837e27daffd"
format_stamp: "Formatted at 2025-06-23 08:04:20 on dist-test-slave-bxhn"
I20250623 08:04:20.421654 32299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 08:04:20.446055 32299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 08:04:20.446483 32270 catalog_manager.cc:1261] Loaded cluster ID: bde910f56f35432e94ef19d006edb93c
I20250623 08:04:20.446569 32270 catalog_manager.cc:1554] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: loading cluster ID for follower catalog manager: success
I20250623 08:04:20.446974 32299 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 08:04:20.447219 32299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 08:04:20.448179 32299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 08:04:20.449515 32270 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: acquiring CA information for follower catalog manager: success
I20250623 08:04:20.449815 32299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 08:04:20.449914 32299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:20.450023 32299 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 08:04:20.450101 32299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:20.451504 32270 catalog_manager.cc:1604] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250623 08:04:20.483265 32299 rpc_server.cc:307] RPC server started. Bound to: 127.31.69.1:34877
I20250623 08:04:20.483340 32425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.69.1:34877 every 8 connection(s)
I20250623 08:04:20.484931 32299 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/data/info.pb
I20250623 08:04:20.489423 32020 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskJxinPB/build/asan/bin/kudu as pid 32299
I20250623 08:04:20.489626 32020 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/instance
I20250623 08:04:20.496694 32020 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskJxinPB/build/asan/bin/kudu
/tmp/dist-test-taskJxinPB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.69.2:0
--local_ip_for_outbound_sockets=127.31.69.2
--webserver_interface=127.31.69.2
--webserver_port=0
--tserver_master_addrs=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--builtin_ntp_servers=127.31.69.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250623 08:04:20.528338 32426 heartbeater.cc:344] Connected to a master server at 127.31.69.60:39691
I20250623 08:04:20.528594 32437 raft_consensus.cc:491] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6644af0f4b2a4b998282b021ec1dc678)
I20250623 08:04:20.528786 32426 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:20.528795 32437 raft_consensus.cc:513] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:20.529947 32426 heartbeater.cc:507] Master 127.31.69.60:39691 requested a full tablet report, sending...
I20250623 08:04:20.530834 32428 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:20.530779 32139 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31835ec4ad74332a00998d431492466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
I20250623 08:04:20.530961 32428 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:20.531423 32428 heartbeater.cc:507] Master 127.31.69.61:33313 requested a full tablet report, sending...
I20250623 08:04:20.532353 32211 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31835ec4ad74332a00998d431492466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6644af0f4b2a4b998282b021ec1dc678" is_pre_election: true
I20250623 08:04:20.533186 32130 ts_manager.cc:194] Registered new tserver with Master: 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:20.534195 32201 ts_manager.cc:194] Registered new tserver with Master: 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:20.534806 32043 leader_election.cc:304] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d31835ec4ad74332a00998d431492466; no voters: 6644af0f4b2a4b998282b021ec1dc678, 9752155c20ee42fbb7ff7b7d5b682444
I20250623 08:04:20.536376 32437 leader_election.cc:290] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:20.536509 32201 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.69.1:53787
I20250623 08:04:20.536549 32437 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:20.539337 32427 heartbeater.cc:344] Connected to a master server at 127.31.69.62:39465
I20250623 08:04:20.539463 32427 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:20.539819 32427 heartbeater.cc:507] Master 127.31.69.62:39465 requested a full tablet report, sending...
I20250623 08:04:20.541193 32059 ts_manager.cc:194] Registered new tserver with Master: 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:20.665369 32284 catalog_manager.cc:1261] Loaded cluster ID: bde910f56f35432e94ef19d006edb93c
I20250623 08:04:20.665472 32284 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: loading cluster ID for follower catalog manager: success
I20250623 08:04:20.668641 32284 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: acquiring CA information for follower catalog manager: success
I20250623 08:04:20.670347 32284 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250623 08:04:20.837261 32434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 08:04:20.837556 32434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 08:04:20.837702 32434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 08:04:20.845623 32434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 08:04:20.845860 32434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.69.2
I20250623 08:04:20.855139 32434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.69.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.69.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.69.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.69.2
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 07:50:47 UTC on 24a791456cd2
build id 6728
ASAN enabled
I20250623 08:04:20.856712 32434 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 08:04:20.858556 32434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 08:04:20.869997 32443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:20.870029 32444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:20.870605 32446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:20.871026 32434 server_base.cc:1048] running on GCE node
I20250623 08:04:20.871553 32434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 08:04:20.872447 32434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 08:04:20.873656 32434 hybrid_clock.cc:648] HybridClock initialized: now 1750665860873607 us; error 43 us; skew 500 ppm
I20250623 08:04:20.873919 32434 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 08:04:20.876154 32434 webserver.cc:469] Webserver started at http://127.31.69.2:38757/ using document root <none> and password file <none>
I20250623 08:04:20.876852 32434 fs_manager.cc:362] Metadata directory not provided
I20250623 08:04:20.876974 32434 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 08:04:20.877379 32434 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 08:04:20.879859 32434 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/data/instance:
uuid: "59f7c614254f42009306b13337f06576"
format_stamp: "Formatted at 2025-06-23 08:04:20 on dist-test-slave-bxhn"
I20250623 08:04:20.880607 32434 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/instance:
uuid: "59f7c614254f42009306b13337f06576"
format_stamp: "Formatted at 2025-06-23 08:04:20 on dist-test-slave-bxhn"
I20250623 08:04:20.885773 32434 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250623 08:04:20.889151 32452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:20.890547 32434 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:20.890679 32434 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal
uuid: "59f7c614254f42009306b13337f06576"
format_stamp: "Formatted at 2025-06-23 08:04:20 on dist-test-slave-bxhn"
I20250623 08:04:20.890899 32434 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 08:04:20.912201 32434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 08:04:20.913096 32434 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 08:04:20.913353 32434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 08:04:20.914342 32434 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 08:04:20.916222 32434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 08:04:20.916311 32434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:20.916415 32434 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 08:04:20.916496 32434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:20.939559 32528 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6644af0f4b2a4b998282b021ec1dc678)
I20250623 08:04:20.939860 32528 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:20.941440 32528 leader_election.cc:290] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:20.942076 32069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9752155c20ee42fbb7ff7b7d5b682444" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d31835ec4ad74332a00998d431492466" is_pre_election: true
I20250623 08:04:20.942082 32211 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9752155c20ee42fbb7ff7b7d5b682444" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6644af0f4b2a4b998282b021ec1dc678" is_pre_election: true
I20250623 08:04:20.943034 32114 leader_election.cc:304] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9752155c20ee42fbb7ff7b7d5b682444; no voters: 6644af0f4b2a4b998282b021ec1dc678, d31835ec4ad74332a00998d431492466
I20250623 08:04:20.943447 32528 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:20.952211 32434 rpc_server.cc:307] RPC server started. Bound to: 127.31.69.2:40197
I20250623 08:04:20.952272 32565 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.69.2:40197 every 8 connection(s)
I20250623 08:04:20.953619 32434 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/data/info.pb
I20250623 08:04:20.961663 32020 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskJxinPB/build/asan/bin/kudu as pid 32434
I20250623 08:04:20.961879 32020 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/instance
I20250623 08:04:20.967265 32020 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskJxinPB/build/asan/bin/kudu
/tmp/dist-test-taskJxinPB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.69.3:0
--local_ip_for_outbound_sockets=127.31.69.3
--webserver_interface=127.31.69.3
--webserver_port=0
--tserver_master_addrs=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--builtin_ntp_servers=127.31.69.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250623 08:04:20.970994 32568 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:20.970945 32567 heartbeater.cc:344] Connected to a master server at 127.31.69.62:39465
I20250623 08:04:20.971421 32567 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:20.971858 32568 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:20.972656 32568 heartbeater.cc:507] Master 127.31.69.61:33313 requested a full tablet report, sending...
I20250623 08:04:20.972680 32567 heartbeater.cc:507] Master 127.31.69.62:39465 requested a full tablet report, sending...
I20250623 08:04:20.974381 32130 ts_manager.cc:194] Registered new tserver with Master: 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:20.974381 32059 ts_manager.cc:194] Registered new tserver with Master: 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:20.974661 32566 heartbeater.cc:344] Connected to a master server at 127.31.69.60:39691
I20250623 08:04:20.974809 32566 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:20.975373 32566 heartbeater.cc:507] Master 127.31.69.60:39691 requested a full tablet report, sending...
I20250623 08:04:20.976326 32201 ts_manager.cc:194] Registered new tserver with Master: 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:20.977452 32201 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.69.2:56751
I20250623 08:04:21.039518 32426 heartbeater.cc:499] Master 127.31.69.60:39691 was elected leader, sending a full tablet report...
I20250623 08:04:21.127202 32574 raft_consensus.cc:491] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6644af0f4b2a4b998282b021ec1dc678)
I20250623 08:04:21.127381 32574 raft_consensus.cc:513] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:21.128051 32574 leader_election.cc:290] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:21.128633 32211 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31835ec4ad74332a00998d431492466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6644af0f4b2a4b998282b021ec1dc678" is_pre_election: true
I20250623 08:04:21.128630 32139 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31835ec4ad74332a00998d431492466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
I20250623 08:04:21.129215 32045 leader_election.cc:304] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d31835ec4ad74332a00998d431492466; no voters: 6644af0f4b2a4b998282b021ec1dc678, 9752155c20ee42fbb7ff7b7d5b682444
I20250623 08:04:21.129542 32574 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 08:04:21.289831 32573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 08:04:21.290158 32573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 08:04:21.290246 32573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250623 08:04:21.298173 32573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 08:04:21.298375 32573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.69.3
I20250623 08:04:21.307799 32573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.69.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.69.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.69.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.69.3
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 07:50:47 UTC on 24a791456cd2
build id 6728
ASAN enabled
I20250623 08:04:21.309335 32573 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 08:04:21.311092 32573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250623 08:04:21.322561 32582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:21.322461 32580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250623 08:04:21.322461 32579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:21.323541 32573 server_base.cc:1048] running on GCE node
I20250623 08:04:21.324075 32573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 08:04:21.324928 32573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 08:04:21.326159 32573 hybrid_clock.cc:648] HybridClock initialized: now 1750665861326118 us; error 31 us; skew 500 ppm
I20250623 08:04:21.326481 32573 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 08:04:21.328737 32573 webserver.cc:469] Webserver started at http://127.31.69.3:37101/ using document root <none> and password file <none>
I20250623 08:04:21.329577 32573 fs_manager.cc:362] Metadata directory not provided
I20250623 08:04:21.329705 32573 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 08:04:21.330329 32573 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250623 08:04:21.334057 32573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/data/instance:
uuid: "125745849172449ba4be8244d61fdcb3"
format_stamp: "Formatted at 2025-06-23 08:04:21 on dist-test-slave-bxhn"
I20250623 08:04:21.335374 32573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/instance:
uuid: "125745849172449ba4be8244d61fdcb3"
format_stamp: "Formatted at 2025-06-23 08:04:21 on dist-test-slave-bxhn"
I20250623 08:04:21.343725 32573 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.000s sys 0.007s
I20250623 08:04:21.348609 32588 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:21.359216 32573 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20250623 08:04:21.359422 32573 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal
uuid: "125745849172449ba4be8244d61fdcb3"
format_stamp: "Formatted at 2025-06-23 08:04:21 on dist-test-slave-bxhn"
I20250623 08:04:21.359687 32573 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 08:04:21.375902 32573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 08:04:21.376709 32573 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 08:04:21.376957 32573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 08:04:21.377885 32573 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250623 08:04:21.379536 32573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250623 08:04:21.379630 32573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:21.379739 32573 ts_tablet_manager.cc:610] Registered 0 tablets
I20250623 08:04:21.379815 32573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:21.413077 32573 rpc_server.cc:307] RPC server started. Bound to: 127.31.69.3:35225
I20250623 08:04:21.413129 32700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.69.3:35225 every 8 connection(s)
I20250623 08:04:21.416260 32573 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/data/info.pb
I20250623 08:04:21.417925 32020 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskJxinPB/build/asan/bin/kudu as pid 32573
I20250623 08:04:21.418103 32020 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/instance
I20250623 08:04:21.435117 32703 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:21.435523 32703 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:21.436415 32703 heartbeater.cc:507] Master 127.31.69.61:33313 requested a full tablet report, sending...
I20250623 08:04:21.437259 32701 heartbeater.cc:344] Connected to a master server at 127.31.69.60:39691
I20250623 08:04:21.437297 32702 heartbeater.cc:344] Connected to a master server at 127.31.69.62:39465
I20250623 08:04:21.437407 32702 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:21.437412 32701 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:21.438014 32701 heartbeater.cc:507] Master 127.31.69.60:39691 requested a full tablet report, sending...
I20250623 08:04:21.438100 32702 heartbeater.cc:507] Master 127.31.69.62:39465 requested a full tablet report, sending...
I20250623 08:04:21.438730 32130 ts_manager.cc:194] Registered new tserver with Master: 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:21.439080 32201 ts_manager.cc:194] Registered new tserver with Master: 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:21.439195 32059 ts_manager.cc:194] Registered new tserver with Master: 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:21.440047 32201 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.69.3:38213
I20250623 08:04:21.444384 32020 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250623 08:04:21.473446 32200 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_479438c7155945b3a31a6658e04b8166"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:21.473510 32201 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_12600ee778e44e5d9ede8fcb5cb93bfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:21.473932 32197 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_62aca081236c4497a7b237b2e9c0c130"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:21.475222 32200 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_479438c7155945b3a31a6658e04b8166 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:21.475314 32196 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_272df10611e5421282bc97a0e50c1a91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:21.475960 32196 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_272df10611e5421282bc97a0e50c1a91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:21.475908 32195 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_0c9dd77d8f934adfb31892904b55d487"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:21.476564 32195 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c9dd77d8f934adfb31892904b55d487 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:21.476604 32198 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_7ce7943b299b4042a67d3a0479d74aef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:21.477188 32198 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ce7943b299b4042a67d3a0479d74aef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:21.478471 32197 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62aca081236c4497a7b237b2e9c0c130 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:21.479168 32194 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_38953562436244d9a1eb0dc32ce8624d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:21.479807 32194 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38953562436244d9a1eb0dc32ce8624d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:21.479982 32566 heartbeater.cc:499] Master 127.31.69.60:39691 was elected leader, sending a full tablet report...
W20250623 08:04:21.481691 32201 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12600ee778e44e5d9ede8fcb5cb93bfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:21.487244 32192 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_62214ed95a2e4eb4b2106354ba900ef7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:21.487838 32192 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62214ed95a2e4eb4b2106354ba900ef7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:21.491276 32193 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_8568488efb53488190aa926892e9f888"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:21.491834 32193 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8568488efb53488190aa926892e9f888 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:21.493113 32199 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "default.Table_d3a13330ee3b4811b83b3a53a8eecde8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:21.493731 32199 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3a13330ee3b4811b83b3a53a8eecde8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:21.527778 32196 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_272df10611e5421282bc97a0e50c1a91" } new_table_name: "default.Table_dd60bd988c9d4b0e9cde6d9d354457c1" modify_external_catalogs: true
I20250623 08:04:21.528227 32198 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_62aca081236c4497a7b237b2e9c0c130" } new_table_name: "default.Table_0a4570ccade4415e92301d381e449f5a" modify_external_catalogs: true
I20250623 08:04:21.529181 32195 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_0c9dd77d8f934adfb31892904b55d487" } new_table_name: "default.Table_c432b187339d44afa3dbb5c0a5bf5067" modify_external_catalogs: true
I20250623 08:04:21.530786 32192 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_479438c7155945b3a31a6658e04b8166" } modify_external_catalogs: true
I20250623 08:04:21.530969 32192 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_479438c7155945b3a31a6658e04b8166" } modify_external_catalogs: true
I20250623 08:04:21.532326 32200 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_38953562436244d9a1eb0dc32ce8624d" } new_table_name: "default.Table_5cbfced11e7c40d098cd1b260a2bbda1" modify_external_catalogs: true
I20250623 08:04:21.527894 32197 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_7ce7943b299b4042a67d3a0479d74aef" } new_table_name: "default.Table_985d82a3508246c7bec42eac00b838f0" modify_external_catalogs: true
W20250623 08:04:21.537429 32195 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b4dd09206454bb79455a0fb158100e0 consensus state has no leader
W20250623 08:04:21.544811 32196 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d93e77ee2e5496da42277f09a5d1afb has no consensus state
I20250623 08:04:21.565548 32195 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_62214ed95a2e4eb4b2106354ba900ef7" } modify_external_catalogs: true
I20250623 08:04:21.565456 32196 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_12600ee778e44e5d9ede8fcb5cb93bfd" } modify_external_catalogs: true
I20250623 08:04:21.565712 32195 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_62214ed95a2e4eb4b2106354ba900ef7" } modify_external_catalogs: true
I20250623 08:04:21.565716 32196 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_12600ee778e44e5d9ede8fcb5cb93bfd" } modify_external_catalogs: true
I20250623 08:04:21.567239 32199 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_dd60bd988c9d4b0e9cde6d9d354457c1" } modify_external_catalogs: true
I20250623 08:04:21.567440 32199 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_dd60bd988c9d4b0e9cde6d9d354457c1" } modify_external_catalogs: true
W20250623 08:04:21.569823 32186 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 6b4dd09206454bb79455a0fb158100e0 (table default.Table_c432b187339d44afa3dbb5c0a5bf5067 [id=acf0055eb47646b2be6079f6c07814c0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6b4dd09206454bb79455a0fb158100e0 consensus state has no leader
W20250623 08:04:21.570665 32197 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6ee90c0361c491b94ca9ff06962de1b has no consensus state
W20250623 08:04:21.571254 32200 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0fc9178845f4d68afcd54c430664c4c has no consensus state
I20250623 08:04:21.571868 32192 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 18a5ae33ea01451a9a5dd301a4858b93
I20250623 08:04:21.580721 32193 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_c432b187339d44afa3dbb5c0a5bf5067" } modify_external_catalogs: true
I20250623 08:04:21.580981 32193 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47922:
table { table_name: "default.Table_c432b187339d44afa3dbb5c0a5bf5067" } modify_external_catalogs: true
W20250623 08:04:21.587359 32198 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c5e77c518e646e19e67c84a8f687801 has no consensus state
I20250623 08:04:21.589568 32199 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 4d93e77ee2e5496da42277f09a5d1afb
I20250623 08:04:21.590441 32195 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet d9a6fbb12e3c42f49ed2200e58dfa2bc
I20250623 08:04:21.591565 32196 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7ebc4bee304e4e2da7754ae9d3e21a03
I20250623 08:04:21.592718 32020 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskJxinPB/build/asan/bin/kudu pid 32167
I20250623 08:04:21.594295 32231 generic_service.cc:196] Checking for leaks (request via RPC)
I20250623 08:04:21.608161 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 6b4dd09206454bb79455a0fb158100e0 (DEFAULT_TABLE table=default.Table_0c9dd77d8f934adfb31892904b55d487 [id=acf0055eb47646b2be6079f6c07814c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:21.610100 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b4dd09206454bb79455a0fb158100e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:21.627369 303 tablet_bootstrap.cc:492] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:21.631242 303 tablet_bootstrap.cc:654] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:21.632702 303 log.cc:826] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: Log is configured to *not* fsync() on all Append() calls
I20250623 08:04:21.635775 303 tablet_bootstrap.cc:492] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:21.636067 303 ts_tablet_manager.cc:1397] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20250623 08:04:21.643190 303 raft_consensus.cc:357] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:21.643486 303 raft_consensus.cc:383] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:21.643625 303 raft_consensus.cc:738] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:21.644187 303 consensus_queue.cc:260] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:21.645848 303 ts_tablet_manager.cc:1428] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
W20250623 08:04:21.707284 32569 tablet.cc:2378] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 08:04:21.736683 32195 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b4dd09206454bb79455a0fb158100e0 from {username='slave'} at 127.0.0.1:47922
I20250623 08:04:21.737084 32020 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskJxinPB/build/asan/bin/kudu with pid 32167
W20250623 08:04:21.759948 32566 heartbeater.cc:646] Failed to heartbeat to 127.31.69.60:39691 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.31.69.60:39691 (error 108)
W20250623 08:04:21.760375 32765 negotiation.cc:337] Failed RPC negotiation. Trace:
0623 08:04:21.579952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.31.69.1:34877)
0623 08:04:21.624394 (+ 44442us) server_negotiation.cc:206] Beginning negotiation
0623 08:04:21.624404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0623 08:04:21.624437 (+ 33us) server_negotiation.cc:407] Connection header received
0623 08:04:21.624546 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0623 08:04:21.624563 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0623 08:04:21.624824 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0623 08:04:21.625043 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0623 08:04:21.759317 (+134274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57920: BlockingRecv error: recv got EOF from 127.0.0.1:57920 (error 108)
Metrics: {"server-negotiator.queue_time_us":44283,"thread_start_us":44169,"threads_started":1}
W20250623 08:04:21.760673 300 negotiation.cc:337] Failed RPC negotiation. Trace:
0623 08:04:21.593420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.31.69.3:35225)
0623 08:04:21.595787 (+ 2367us) server_negotiation.cc:206] Beginning negotiation
0623 08:04:21.595796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0623 08:04:21.595832 (+ 36us) server_negotiation.cc:407] Connection header received
0623 08:04:21.595934 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0623 08:04:21.595942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0623 08:04:21.596207 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0623 08:04:21.596579 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0623 08:04:21.737403 (+140824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0623 08:04:21.739553 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0623 08:04:21.759317 (+ 19764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46262: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":2209,"thread_start_us":1932,"threads_started":1}
W20250623 08:04:21.760933 32708 connection.cc:537] client connection to 127.31.69.60:39691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:21.761204 32114 connection.cc:537] server connection from 127.0.0.1:50416 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:21.761976 32708 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.69.60:39691)
W20250623 08:04:21.762830 32708 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.31.69.60:39691)
W20250623 08:04:21.763427 32708 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.31.69.60:39691)
W20250623 08:04:21.765446 32708 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.31.69.60:39691)
I20250623 08:04:21.942348 32701 heartbeater.cc:499] Master 127.31.69.60:39691 was elected leader, sending a full tablet report...
W20250623 08:04:21.943574 32701 heartbeater.cc:646] Failed to heartbeat to 127.31.69.60:39691 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
W20250623 08:04:22.044435 32426 heartbeater.cc:646] Failed to heartbeat to 127.31.69.60:39691 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:22.101238 309 raft_consensus.cc:491] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6644af0f4b2a4b998282b021ec1dc678)
I20250623 08:04:22.101469 309 raft_consensus.cc:513] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:22.102093 309 leader_election.cc:290] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:22.102530 32139 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31835ec4ad74332a00998d431492466" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
W20250623 08:04:22.104018 32043 leader_election.cc:336] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691): Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:22.104153 32043 leader_election.cc:304] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d31835ec4ad74332a00998d431492466; no voters: 6644af0f4b2a4b998282b021ec1dc678, 9752155c20ee42fbb7ff7b7d5b682444
I20250623 08:04:22.104389 309 raft_consensus.cc:2747] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:22.238962 312 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6644af0f4b2a4b998282b021ec1dc678)
I20250623 08:04:22.239131 312 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:22.239884 312 leader_election.cc:290] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:22.240327 32069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9752155c20ee42fbb7ff7b7d5b682444" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "d31835ec4ad74332a00998d431492466" is_pre_election: true
I20250623 08:04:22.240562 32069 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9752155c20ee42fbb7ff7b7d5b682444 in term 1.
I20250623 08:04:22.241235 32114 leader_election.cc:304] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9752155c20ee42fbb7ff7b7d5b682444, d31835ec4ad74332a00998d431492466; no voters:
I20250623 08:04:22.241513 312 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:22.241642 312 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6644af0f4b2a4b998282b021ec1dc678)
I20250623 08:04:22.241761 312 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 1 FOLLOWER]: Advancing to term 2
W20250623 08:04:22.242007 32114 leader_election.cc:336] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691): Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:22.243863 312 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:22.244508 312 leader_election.cc:290] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 2 election: Requested vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:22.244962 32069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9752155c20ee42fbb7ff7b7d5b682444" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "d31835ec4ad74332a00998d431492466"
I20250623 08:04:22.245179 32069 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 1 FOLLOWER]: Advancing to term 2
W20250623 08:04:22.245713 32114 leader_election.cc:336] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691): Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111)
I20250623 08:04:22.247380 32069 raft_consensus.cc:2466] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9752155c20ee42fbb7ff7b7d5b682444 in term 2.
I20250623 08:04:22.247822 32114 leader_election.cc:304] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9752155c20ee42fbb7ff7b7d5b682444, d31835ec4ad74332a00998d431492466; no voters: 6644af0f4b2a4b998282b021ec1dc678
I20250623 08:04:22.248682 312 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:22.249051 312 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 2 LEADER]: Becoming Leader. State: Replica: 9752155c20ee42fbb7ff7b7d5b682444, State: Running, Role: LEADER
I20250623 08:04:22.249464 312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:22.250604 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9752155c20ee42fbb7ff7b7d5b682444. Latest consensus state: current_term: 2 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:22.250831 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: LEADER
I20250623 08:04:22.512229 319 raft_consensus.cc:491] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:22.512503 319 raft_consensus.cc:513] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:22.514138 319 leader_election.cc:290] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:22.522562 32453 leader_election.cc:343] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 6b4dd09206454bb79455a0fb158100e0
W20250623 08:04:22.522847 32453 leader_election.cc:343] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 6b4dd09206454bb79455a0fb158100e0
I20250623 08:04:22.522977 32453 leader_election.cc:304] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:22.523321 319 raft_consensus.cc:2747] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:22.603384 32020 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskJxinPB/build/asan/bin/kudu
/tmp/dist-test-taskJxinPB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.60:39691
--webserver_interface=127.31.69.60
--webserver_port=35145
--builtin_ntp_servers=127.31.69.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250623 08:04:22.671049 32069 raft_consensus.cc:1273] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 2 FOLLOWER]: Refusing update from remote peer 9752155c20ee42fbb7ff7b7d5b682444: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250623 08:04:22.671756 315 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250623 08:04:22.673840 325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9752155c20ee42fbb7ff7b7d5b682444. Latest consensus state: current_term: 2 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:22.674196 325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:22.676636 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:22.676927 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: LEADER
W20250623 08:04:22.678947 32114 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 -> Peer 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691): Couldn't send request to peer 6644af0f4b2a4b998282b021ec1dc678. Status: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:22.679999 325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:22.680222 325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:22.767350 316 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 08:04:22.770001 316 catalog_manager.cc:671] Loaded metadata for table default.Table_d3a13330ee3b4811b83b3a53a8eecde8 [id=1cac605ff7f8488dbb5c2f95fc31dec2]
I20250623 08:04:22.770419 316 catalog_manager.cc:671] Loaded metadata for table default.Table_5cbfced11e7c40d098cd1b260a2bbda1 [id=2fd444ee4cf340118edab36597dc75f8]
I20250623 08:04:22.770790 316 catalog_manager.cc:671] Loaded metadata for table default.Table_8568488efb53488190aa926892e9f888 [id=5630a19db2d046d6b7e94f8d724a13ab]
I20250623 08:04:22.771432 316 catalog_manager.cc:671] Loaded metadata for table default.Table_0a4570ccade4415e92301d381e449f5a [id=9a792e7fecc84b6dab8eeced687eed5d]
I20250623 08:04:22.771920 316 catalog_manager.cc:671] Loaded metadata for table default.Table_985d82a3508246c7bec42eac00b838f0 [id=c967f23f2af84202a59d895e5fb7a2de]
I20250623 08:04:22.773572 316 tablet_loader.cc:96] loaded metadata for tablet 3c5e77c518e646e19e67c84a8f687801 (table default.Table_0a4570ccade4415e92301d381e449f5a [id=9a792e7fecc84b6dab8eeced687eed5d])
I20250623 08:04:22.774375 316 tablet_loader.cc:96] loaded metadata for tablet b0fc9178845f4d68afcd54c430664c4c (table default.Table_5cbfced11e7c40d098cd1b260a2bbda1 [id=2fd444ee4cf340118edab36597dc75f8])
I20250623 08:04:22.774660 316 tablet_loader.cc:96] loaded metadata for tablet b6ee90c0361c491b94ca9ff06962de1b (table default.Table_985d82a3508246c7bec42eac00b838f0 [id=c967f23f2af84202a59d895e5fb7a2de])
I20250623 08:04:22.774935 316 tablet_loader.cc:96] loaded metadata for tablet bf435a89f4f44a299b0ef5f4fc3891e6 (table default.Table_8568488efb53488190aa926892e9f888 [id=5630a19db2d046d6b7e94f8d724a13ab])
I20250623 08:04:22.775179 316 tablet_loader.cc:96] loaded metadata for tablet c7d66b3414784129a5fd5941860666cd (table default.Table_d3a13330ee3b4811b83b3a53a8eecde8 [id=1cac605ff7f8488dbb5c2f95fc31dec2])
I20250623 08:04:22.775553 316 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 08:04:22.776544 316 catalog_manager.cc:1261] Loaded cluster ID: bde910f56f35432e94ef19d006edb93c
I20250623 08:04:22.776643 316 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 08:04:22.777933 316 catalog_manager.cc:1506] Loading token signing keys...
I20250623 08:04:22.778841 316 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Loaded TSK: 0
I20250623 08:04:22.779727 316 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250623 08:04:22.789803 32123 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b4dd09206454bb79455a0fb158100e0 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:22.790004 32121 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b4dd09206454bb79455a0fb158100e0 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:22.789855 32122 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b4dd09206454bb79455a0fb158100e0 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:22.791002 32123 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b4dd09206454bb79455a0fb158100e0 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:22.791503 32123 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b4dd09206454bb79455a0fb158100e0 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:22.792748 32121 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_5cbfced11e7c40d098cd1b260a2bbda1" } new_table_name: "default.Table_a8617f1142ba4aa9adf5a1718a411d05" modify_external_catalogs: true
I20250623 08:04:22.792352 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_4e025d143fd241f5a3368980740a8963"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:22.791412 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_5407838a4fc64b35bc677c8949d65491"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:22.792095 32123 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c432b187339d44afa3dbb5c0a5bf5067" } modify_external_catalogs: true
I20250623 08:04:22.797865 32123 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c432b187339d44afa3dbb5c0a5bf5067" } modify_external_catalogs: true
W20250623 08:04:22.798805 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e025d143fd241f5a3368980740a8963 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:22.789914 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_13b0e15543044cd58b65937ca08d8df4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:22.798692 32123 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_0a4570ccade4415e92301d381e449f5a" } new_table_name: "default.Table_25af89afee014e4bbecdb22a93780edc" modify_external_catalogs: true
I20250623 08:04:22.792062 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_8840b19d751a4c2da75bcdc6da8dd0a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:22.800987 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8840b19d751a4c2da75bcdc6da8dd0a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:22.791736 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_edea5b6ed0234c4bbe5755a9e1c6f13e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:22.801589 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edea5b6ed0234c4bbe5755a9e1c6f13e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:22.791055 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_eb1d2a068cde47b1b3cb9cd7bb92bfe8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:22.802984 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb1d2a068cde47b1b3cb9cd7bb92bfe8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:22.792925 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_b7f62bf95f5843e8a1e9e0a08386bf06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:22.804894 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7f62bf95f5843e8a1e9e0a08386bf06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:22.804981 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13b0e15543044cd58b65937ca08d8df4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:22.806087 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5407838a4fc64b35bc677c8949d65491 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:22.794203 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_aeb7585a9aad4577b59b1b1164aa016e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:22.807750 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeb7585a9aad4577b59b1b1164aa016e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:22.810897 32121 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0fc9178845f4d68afcd54c430664c4c has no consensus state
I20250623 08:04:22.813578 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_76d7b801c99542fb9d0f640ee5d549bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:22.814136 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76d7b801c99542fb9d0f640ee5d549bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:22.819727 32123 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c5e77c518e646e19e67c84a8f687801 has no consensus state
I20250623 08:04:22.827713 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_05daabd5e8b848d49be576c05ed72c23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:22.828047 32129 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d3a13330ee3b4811b83b3a53a8eecde8" } new_table_name: "default.Table_10f030b47f8e4b31ba84115caef0203d" modify_external_catalogs: true
W20250623 08:04:22.828358 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05daabd5e8b848d49be576c05ed72c23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:22.829543 32128 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8568488efb53488190aa926892e9f888" } new_table_name: "default.Table_097551427ef74c7691bb08662c2c088c" modify_external_catalogs: true
I20250623 08:04:22.829815 32127 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_985d82a3508246c7bec42eac00b838f0" } modify_external_catalogs: true
I20250623 08:04:22.829998 32127 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_985d82a3508246c7bec42eac00b838f0" } modify_external_catalogs: true
I20250623 08:04:22.830441 32125 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a8617f1142ba4aa9adf5a1718a411d05" } new_table_name: "default.Table_52580ab1940b40b0b31f4b95f9993f67" modify_external_catalogs: true
I20250623 08:04:22.848412 32121 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_edea5b6ed0234c4bbe5755a9e1c6f13e" } modify_external_catalogs: true
I20250623 08:04:22.848675 32124 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_4e025d143fd241f5a3368980740a8963" } modify_external_catalogs: true
I20250623 08:04:22.848902 32124 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_4e025d143fd241f5a3368980740a8963" } modify_external_catalogs: true
I20250623 08:04:22.849900 32126 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8840b19d751a4c2da75bcdc6da8dd0a8" } modify_external_catalogs: true
I20250623 08:04:22.850107 32126 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8840b19d751a4c2da75bcdc6da8dd0a8" } modify_external_catalogs: true
I20250623 08:04:22.848701 32121 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_edea5b6ed0234c4bbe5755a9e1c6f13e" } modify_external_catalogs: true
I20250623 08:04:22.851416 32122 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_25af89afee014e4bbecdb22a93780edc" } modify_external_catalogs: true
I20250623 08:04:22.851612 32122 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_25af89afee014e4bbecdb22a93780edc" } modify_external_catalogs: true
W20250623 08:04:22.853354 32116 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet b0fc9178845f4d68afcd54c430664c4c (table default.Table_a8617f1142ba4aa9adf5a1718a411d05 [id=2fd444ee4cf340118edab36597dc75f8], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet b0fc9178845f4d68afcd54c430664c4c consensus state has no leader
W20250623 08:04:22.859634 32129 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7d66b3414784129a5fd5941860666cd consensus state has no leader
W20250623 08:04:22.860512 32128 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf435a89f4f44a299b0ef5f4fc3891e6 consensus state has no leader
W20250623 08:04:22.861819 32125 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0fc9178845f4d68afcd54c430664c4c consensus state has no leader
I20250623 08:04:22.872287 32359 tablet_service.cc:1468] Processing CreateTablet for tablet c7d66b3414784129a5fd5941860666cd (DEFAULT_TABLE table=default.Table_d3a13330ee3b4811b83b3a53a8eecde8 [id=1cac605ff7f8488dbb5c2f95fc31dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.873106 32361 tablet_service.cc:1468] Processing CreateTablet for tablet b6ee90c0361c491b94ca9ff06962de1b (DEFAULT_TABLE table=default.Table_985d82a3508246c7bec42eac00b838f0 [id=c967f23f2af84202a59d895e5fb7a2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.872339 32358 tablet_service.cc:1468] Processing CreateTablet for tablet b0fc9178845f4d68afcd54c430664c4c (DEFAULT_TABLE table=default.Table_a8617f1142ba4aa9adf5a1718a411d05 [id=2fd444ee4cf340118edab36597dc75f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.874018 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6ee90c0361c491b94ca9ff06962de1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.874477 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fc9178845f4d68afcd54c430664c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.874994 32360 tablet_service.cc:1468] Processing CreateTablet for tablet 3c5e77c518e646e19e67c84a8f687801 (DEFAULT_TABLE table=default.Table_25af89afee014e4bbecdb22a93780edc [id=9a792e7fecc84b6dab8eeced687eed5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.875557 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5e77c518e646e19e67c84a8f687801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.872238 32357 tablet_service.cc:1468] Processing CreateTablet for tablet bf435a89f4f44a299b0ef5f4fc3891e6 (DEFAULT_TABLE table=default.Table_8568488efb53488190aa926892e9f888 [id=5630a19db2d046d6b7e94f8d724a13ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.877605 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf435a89f4f44a299b0ef5f4fc3891e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.878649 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7d66b3414784129a5fd5941860666cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.896749 32636 tablet_service.cc:1468] Processing CreateTablet for tablet b6ee90c0361c491b94ca9ff06962de1b (DEFAULT_TABLE table=default.Table_985d82a3508246c7bec42eac00b838f0 [id=c967f23f2af84202a59d895e5fb7a2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.898312 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6ee90c0361c491b94ca9ff06962de1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.903291 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 3c5e77c518e646e19e67c84a8f687801 (DEFAULT_TABLE table=default.Table_25af89afee014e4bbecdb22a93780edc [id=9a792e7fecc84b6dab8eeced687eed5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.903764 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5e77c518e646e19e67c84a8f687801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.904763 32634 tablet_service.cc:1468] Processing CreateTablet for tablet c7d66b3414784129a5fd5941860666cd (DEFAULT_TABLE table=default.Table_d3a13330ee3b4811b83b3a53a8eecde8 [id=1cac605ff7f8488dbb5c2f95fc31dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.905225 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7d66b3414784129a5fd5941860666cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.905462 32127 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet b6ee90c0361c491b94ca9ff06962de1b
I20250623 08:04:22.906129 32633 tablet_service.cc:1468] Processing CreateTablet for tablet b0fc9178845f4d68afcd54c430664c4c (DEFAULT_TABLE table=default.Table_a8617f1142ba4aa9adf5a1718a411d05 [id=2fd444ee4cf340118edab36597dc75f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.906551 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fc9178845f4d68afcd54c430664c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.907434 32632 tablet_service.cc:1468] Processing CreateTablet for tablet bf435a89f4f44a299b0ef5f4fc3891e6 (DEFAULT_TABLE table=default.Table_8568488efb53488190aa926892e9f888 [id=5630a19db2d046d6b7e94f8d724a13ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.907581 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ee90c0361c491b94ca9ff06962de1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:22.907847 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf435a89f4f44a299b0ef5f4fc3891e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.908583 32124 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 5c8ad1bdf1174b26bae5aa254a7c61ce
I20250623 08:04:22.909482 32500 tablet_service.cc:1468] Processing CreateTablet for tablet b6ee90c0361c491b94ca9ff06962de1b (DEFAULT_TABLE table=default.Table_985d82a3508246c7bec42eac00b838f0 [id=c967f23f2af84202a59d895e5fb7a2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.909961 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6ee90c0361c491b94ca9ff06962de1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.914723 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ee90c0361c491b94ca9ff06962de1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:22.916683 32121 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet be232a3ecdc349c4b9babe940a0c5cd2
I20250623 08:04:22.919438 32126 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 1a4563ba16f2484ab8a1467c5016673f
I20250623 08:04:22.919878 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 3c5e77c518e646e19e67c84a8f687801
I20250623 08:04:22.931120 339 tablet_bootstrap.cc:492] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:22.934547 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 3c5e77c518e646e19e67c84a8f687801 (DEFAULT_TABLE table=default.Table_25af89afee014e4bbecdb22a93780edc [id=9a792e7fecc84b6dab8eeced687eed5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.935151 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5e77c518e646e19e67c84a8f687801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.935302 32496 tablet_service.cc:1468] Processing CreateTablet for tablet bf435a89f4f44a299b0ef5f4fc3891e6 (DEFAULT_TABLE table=default.Table_8568488efb53488190aa926892e9f888 [id=5630a19db2d046d6b7e94f8d724a13ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.935835 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf435a89f4f44a299b0ef5f4fc3891e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.939129 32497 tablet_service.cc:1468] Processing CreateTablet for tablet b0fc9178845f4d68afcd54c430664c4c (DEFAULT_TABLE table=default.Table_a8617f1142ba4aa9adf5a1718a411d05 [id=2fd444ee4cf340118edab36597dc75f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.939579 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fc9178845f4d68afcd54c430664c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.934561 32498 tablet_service.cc:1468] Processing CreateTablet for tablet c7d66b3414784129a5fd5941860666cd (DEFAULT_TABLE table=default.Table_d3a13330ee3b4811b83b3a53a8eecde8 [id=1cac605ff7f8488dbb5c2f95fc31dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.943593 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ee90c0361c491b94ca9ff06962de1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:22.943645 339 tablet_bootstrap.cc:654] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:22.944545 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7d66b3414784129a5fd5941860666cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.948210 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c5e77c518e646e19e67c84a8f687801 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:22.949362 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c5e77c518e646e19e67c84a8f687801 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:22.957185 342 tablet_bootstrap.cc:492] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:22.959550 342 tablet_bootstrap.cc:654] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:22.964288 32114 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b6ee90c0361c491b94ca9ff06962de1b because the tablet was not found. No further retry: Not found: Tablet not found: b6ee90c0361c491b94ca9ff06962de1b
W20250623 08:04:22.964612 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 3c5e77c518e646e19e67c84a8f687801 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c5e77c518e646e19e67c84a8f687801 already in progress: creating tablet
I20250623 08:04:22.969756 343 tablet_bootstrap.cc:492] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:22.973507 343 tablet_bootstrap.cc:654] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:22.975847 32360 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c5e77c518e646e19e67c84a8f687801 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:22.976364 32495 tablet_service.cc:1468] Processing CreateTablet for tablet 99ea7850738743deb7afa87ea2e40daa (DEFAULT_TABLE table=default.Table_5407838a4fc64b35bc677c8949d65491 [id=6cc9b03b74284f159ac54dd2dc10dc4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.976967 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ea7850738743deb7afa87ea2e40daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.978386 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 5c7d30889c024c84b6585f007504139e (DEFAULT_TABLE table=default.Table_76d7b801c99542fb9d0f640ee5d549bd [id=816747bd5e004ae5964b7b885aa249fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.978796 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c7d30889c024c84b6585f007504139e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.978870 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 3bf5243f44c043a0a1e9bc441fc91e70 (DEFAULT_TABLE table=default.Table_13b0e15543044cd58b65937ca08d8df4 [id=adb0245ebf874699887e167d5b46d1d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.979228 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf5243f44c043a0a1e9bc441fc91e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.979810 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 75dde3beac4e47879c3f5e16151a4675 (DEFAULT_TABLE table=default.Table_eb1d2a068cde47b1b3cb9cd7bb92bfe8 [id=26c5fff9eaea4788844a55e87ada7215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.980168 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75dde3beac4e47879c3f5e16151a4675. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:22.980638 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet b6ee90c0361c491b94ca9ff06962de1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6ee90c0361c491b94ca9ff06962de1b already in progress: creating tablet
W20250623 08:04:22.980911 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 3c5e77c518e646e19e67c84a8f687801 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c5e77c518e646e19e67c84a8f687801 already in progress: creating tablet
W20250623 08:04:22.981155 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet b6ee90c0361c491b94ca9ff06962de1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6ee90c0361c491b94ca9ff06962de1b already in progress: creating tablet
W20250623 08:04:22.981388 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 3c5e77c518e646e19e67c84a8f687801 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c5e77c518e646e19e67c84a8f687801 already in progress: creating tablet
I20250623 08:04:22.982992 32498 tablet_service.cc:1468] Processing CreateTablet for tablet b6e29ab07df249748092a2f50a8c0273 (DEFAULT_TABLE table=default.Table_b7f62bf95f5843e8a1e9e0a08386bf06 [id=e7c90b63c16a4cca9365fd4733ba1296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.983326 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6e29ab07df249748092a2f50a8c0273. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.983783 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 75dde3beac4e47879c3f5e16151a4675 (DEFAULT_TABLE table=default.Table_eb1d2a068cde47b1b3cb9cd7bb92bfe8 [id=26c5fff9eaea4788844a55e87ada7215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.984107 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75dde3beac4e47879c3f5e16151a4675. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.984936 32360 tablet_service.cc:1468] Processing CreateTablet for tablet 99ea7850738743deb7afa87ea2e40daa (DEFAULT_TABLE table=default.Table_5407838a4fc64b35bc677c8949d65491 [id=6cc9b03b74284f159ac54dd2dc10dc4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.985064 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 3bf5243f44c043a0a1e9bc441fc91e70 (DEFAULT_TABLE table=default.Table_13b0e15543044cd58b65937ca08d8df4 [id=adb0245ebf874699887e167d5b46d1d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.985313 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ea7850738743deb7afa87ea2e40daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.985541 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf5243f44c043a0a1e9bc441fc91e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.987185 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 5c7d30889c024c84b6585f007504139e (DEFAULT_TABLE table=default.Table_76d7b801c99542fb9d0f640ee5d549bd [id=816747bd5e004ae5964b7b885aa249fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:22.987592 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c7d30889c024c84b6585f007504139e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.989099 342 tablet_bootstrap.cc:492] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:22.989257 342 ts_tablet_manager.cc:1397] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20250623 08:04:22.990329 342 raft_consensus.cc:357] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:22.990537 342 raft_consensus.cc:383] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:22.990624 342 raft_consensus.cc:738] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:22.990942 342 consensus_queue.cc:260] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:22.991524 342 ts_tablet_manager.cc:1428] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:22.991813 342 tablet_bootstrap.cc:492] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:22.997915 32122 master_service.cc:948] ReplaceTablet: received request to replace tablet bf435a89f4f44a299b0ef5f4fc3891e6 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:22.998003 32568 heartbeater.cc:499] Master 127.31.69.61:33313 was elected leader, sending a full tablet report...
I20250623 08:04:22.998509 32126 master_service.cc:948] ReplaceTablet: received request to replace tablet b0fc9178845f4d68afcd54c430664c4c from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.000851 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ee90c0361c491b94ca9ff06962de1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.001222 344 tablet_replica.cc:331] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:23.001729 342 tablet_bootstrap.cc:654] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.001737 344 raft_consensus.cc:2241] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.002337 344 raft_consensus.cc:2270] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.003330 32121 master_service.cc:948] ReplaceTablet: received request to replace tablet 5c7d30889c024c84b6585f007504139e from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.003602 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet 3c5e77c518e646e19e67c84a8f687801 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.004664 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet bf435a89f4f44a299b0ef5f4fc3891e6 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.007094 32356 tablet_service.cc:1468] Processing CreateTablet for tablet b6e29ab07df249748092a2f50a8c0273 (DEFAULT_TABLE table=default.Table_b7f62bf95f5843e8a1e9e0a08386bf06 [id=e7c90b63c16a4cca9365fd4733ba1296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.007601 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6e29ab07df249748092a2f50a8c0273. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.975621 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 99ea7850738743deb7afa87ea2e40daa (DEFAULT_TABLE table=default.Table_5407838a4fc64b35bc677c8949d65491 [id=6cc9b03b74284f159ac54dd2dc10dc4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.011318 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ea7850738743deb7afa87ea2e40daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.012363 344 ts_tablet_manager.cc:1905] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:22.976701 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 5c7d30889c024c84b6585f007504139e (DEFAULT_TABLE table=default.Table_76d7b801c99542fb9d0f640ee5d549bd [id=816747bd5e004ae5964b7b885aa249fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.015286 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c7d30889c024c84b6585f007504139e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:22.977764 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 3bf5243f44c043a0a1e9bc441fc91e70 (DEFAULT_TABLE table=default.Table_13b0e15543044cd58b65937ca08d8df4 [id=adb0245ebf874699887e167d5b46d1d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.018533 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf5243f44c043a0a1e9bc441fc91e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.020155 342 tablet_bootstrap.cc:492] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.020331 342 ts_tablet_manager.cc:1397] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250623 08:04:23.021081 342 raft_consensus.cc:357] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.021282 342 raft_consensus.cc:383] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.021354 342 raft_consensus.cc:738] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.021620 342 consensus_queue.cc:260] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.024201 32130 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_76d7b801c99542fb9d0f640ee5d549bd" } new_table_name: "default.Table_d27db7e0fea34b5182353dee8ae23cac" modify_external_catalogs: true
I20250623 08:04:22.979147 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 75dde3beac4e47879c3f5e16151a4675 (DEFAULT_TABLE table=default.Table_eb1d2a068cde47b1b3cb9cd7bb92bfe8 [id=26c5fff9eaea4788844a55e87ada7215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.024541 32125 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_05daabd5e8b848d49be576c05ed72c23" } new_table_name: "default.Table_fac8d606d7454d3184af3a8d4c20ccc4" modify_external_catalogs: true
I20250623 08:04:23.024667 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75dde3beac4e47879c3f5e16151a4675. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.025666 342 ts_tablet_manager.cc:1428] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:23.026875 342 tablet_bootstrap.cc:492] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.027909 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_eb7a5a786084417cb64d7093913e0e95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.028470 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb7a5a786084417cb64d7093913e0e95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.028892 339 log.cc:826] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: Log is configured to *not* fsync() on all Append() calls
I20250623 08:04:23.029156 342 tablet_bootstrap.cc:654] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:22.980103 32634 tablet_service.cc:1468] Processing CreateTablet for tablet b6e29ab07df249748092a2f50a8c0273 (DEFAULT_TABLE table=default.Table_b7f62bf95f5843e8a1e9e0a08386bf06 [id=e7c90b63c16a4cca9365fd4733ba1296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.029736 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6e29ab07df249748092a2f50a8c0273. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.035779 344 ts_tablet_manager.cc:1918] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.036010 344 log.cc:1199] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b6ee90c0361c491b94ca9ff06962de1b
I20250623 08:04:23.036583 344 ts_tablet_manager.cc:1939] T b6ee90c0361c491b94ca9ff06962de1b P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:23.038642 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b6ee90c0361c491b94ca9ff06962de1b (table default.Table_985d82a3508246c7bec42eac00b838f0 [id=c967f23f2af84202a59d895e5fb7a2de]) successfully deleted
I20250623 08:04:23.039306 339 tablet_bootstrap.cc:492] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.039677 339 ts_tablet_manager.cc:1397] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.005s
I20250623 08:04:23.039862 342 tablet_bootstrap.cc:492] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.040050 342 ts_tablet_manager.cc:1397] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20250623 08:04:23.041332 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_0beadbd550664534bbf90d51ca7a4b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.041560 342 raft_consensus.cc:357] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.041772 342 raft_consensus.cc:383] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.041726 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_c2bc4709093c46c7a0db851d422b41a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.041859 342 raft_consensus.cc:738] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
W20250623 08:04:23.041957 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0beadbd550664534bbf90d51ca7a4b0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.042104 342 consensus_queue.cc:260] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:23.042289 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2bc4709093c46c7a0db851d422b41a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.043704 342 ts_tablet_manager.cc:1428] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250623 08:04:23.044804 342 tablet_bootstrap.cc:492] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.047015 342 tablet_bootstrap.cc:654] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.048126 32632 tablet_service.cc:1468] Processing CreateTablet for tablet d3a3c5a446184329b55fcc2cbf969831 (DEFAULT_TABLE table=default.Table_aeb7585a9aad4577b59b1b1164aa016e [id=cf9d65382d8b4fd98f18c628c2ccad23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.048259 32127 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b0fc9178845f4d68afcd54c430664c4c (table default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]). Expected version 3 got 0
I20250623 08:04:23.048476 32127 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet bf435a89f4f44a299b0ef5f4fc3891e6 (table default.Table_097551427ef74c7691bb08662c2c088c [id=5630a19db2d046d6b7e94f8d724a13ab]). Expected version 1 got 0
I20250623 08:04:23.048553 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a3c5a446184329b55fcc2cbf969831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.048729 32127 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c7d66b3414784129a5fd5941860666cd (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]). Expected version 1 got 0
I20250623 08:04:23.049866 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 9b5a56fe461641bea2bb269bf0564415 (DEFAULT_TABLE table=default.Table_05daabd5e8b848d49be576c05ed72c23 [id=d94a8f840a0c48a6bdfdca2ff5f0c5dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.050462 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5a56fe461641bea2bb269bf0564415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.053589 32359 tablet_service.cc:1468] Processing CreateTablet for tablet d3a3c5a446184329b55fcc2cbf969831 (DEFAULT_TABLE table=default.Table_aeb7585a9aad4577b59b1b1164aa016e [id=cf9d65382d8b4fd98f18c628c2ccad23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.054085 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a3c5a446184329b55fcc2cbf969831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.055049 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 9b5a56fe461641bea2bb269bf0564415 (DEFAULT_TABLE table=default.Table_05daabd5e8b848d49be576c05ed72c23 [id=d94a8f840a0c48a6bdfdca2ff5f0c5dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.055362 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5a56fe461641bea2bb269bf0564415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.055661 342 tablet_bootstrap.cc:492] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.055812 342 ts_tablet_manager.cc:1397] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20250623 08:04:23.056542 342 raft_consensus.cc:357] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.056733 342 raft_consensus.cc:383] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.056802 342 raft_consensus.cc:738] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.057061 342 consensus_queue.cc:260] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.057621 342 ts_tablet_manager.cc:1428] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:23.057890 342 tablet_bootstrap.cc:492] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.061812 342 tablet_bootstrap.cc:654] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.063217 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 9b5a56fe461641bea2bb269bf0564415 (DEFAULT_TABLE table=default.Table_05daabd5e8b848d49be576c05ed72c23 [id=d94a8f840a0c48a6bdfdca2ff5f0c5dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.063408 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b4dd09206454bb79455a0fb158100e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:21 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.063642 344 tablet_replica.cc:331] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:23.063774 32498 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c5e77c518e646e19e67c84a8f687801 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.063838 344 raft_consensus.cc:2241] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.064019 344 raft_consensus.cc:2270] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.064025 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ee90c0361c491b94ca9ff06962de1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.064568 344 ts_tablet_manager.cc:1905] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.063648 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b5a56fe461641bea2bb269bf0564415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.066810 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_cd6067d4c2734a3e894f926700680ca2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.070502 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd6067d4c2734a3e894f926700680ca2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.078478 344 ts_tablet_manager.cc:1918] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250623 08:04:23.079433 324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 08:04:23.079944 324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20250623 08:04:23.080122 344 log.cc:1199] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/6b4dd09206454bb79455a0fb158100e0
W20250623 08:04:23.080150 324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250623 08:04:23.080538 344 ts_tablet_manager.cc:1939] T 6b4dd09206454bb79455a0fb158100e0 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:23.083103 32130 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c7d30889c024c84b6585f007504139e consensus state has no leader
W20250623 08:04:23.085083 32125 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9b5a56fe461641bea2bb269bf0564415 consensus state has no leader
I20250623 08:04:23.087229 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 6b4dd09206454bb79455a0fb158100e0 (table default.Table_c432b187339d44afa3dbb5c0a5bf5067 [id=acf0055eb47646b2be6079f6c07814c0]) successfully deleted
W20250623 08:04:23.087545 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 3c5e77c518e646e19e67c84a8f687801 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c5e77c518e646e19e67c84a8f687801 already in progress: creating tablet
W20250623 08:04:23.087813 32114 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet b6ee90c0361c491b94ca9ff06962de1b because the tablet was not found. No further retry: Not found: Tablet not found: b6ee90c0361c491b94ca9ff06962de1b
I20250623 08:04:23.089108 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_0e539ce89ea247e8b92fae1cc603ca88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.089632 342 tablet_bootstrap.cc:492] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
W20250623 08:04:23.089684 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e539ce89ea247e8b92fae1cc603ca88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.089887 342 ts_tablet_manager.cc:1397] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.008s
I20250623 08:04:23.090741 342 raft_consensus.cc:357] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.091014 342 raft_consensus.cc:383] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.091152 342 raft_consensus.cc:738] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.091674 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_2ab537bbc9724fd69812910c17dca994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.091807 342 consensus_queue.cc:260] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:23.092157 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ab537bbc9724fd69812910c17dca994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.092615 342 ts_tablet_manager.cc:1428] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:23.092890 342 tablet_bootstrap.cc:492] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:23.093601 324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 08:04:23.093750 324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 08:04:23.093905 324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 08:04:23.094012 324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 08:04:23.094019 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_ced228608de64c2892b15f6dbd5a2d4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.094504 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ced228608de64c2892b15f6dbd5a2d4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.095012 342 tablet_bootstrap.cc:654] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.099152 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_fe1953ea80584690bccd51204d81a314"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.099532 32121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 5c7d30889c024c84b6585f007504139e
W20250623 08:04:23.099661 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe1953ea80584690bccd51204d81a314 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.100131 32121 catalog_manager.cc:6652] ReplaceTablet: tablet 5c7d30889c024c84b6585f007504139e deleted and replaced by tablet b9d871507dcc46fc80f6e0ac739c6476
I20250623 08:04:23.100581 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_d2bb692b2c70474ea4b997cde4fe161a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.101246 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2bb692b2c70474ea4b997cde4fe161a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.102679 32126 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet b0fc9178845f4d68afcd54c430664c4c
I20250623 08:04:23.103188 32126 catalog_manager.cc:6652] ReplaceTablet: tablet b0fc9178845f4d68afcd54c430664c4c deleted and replaced by tablet e0c97148429046dc88213ff70a605c99
I20250623 08:04:23.103605 32126 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_eb1d2a068cde47b1b3cb9cd7bb92bfe8" } new_table_name: "default.Table_3f74ba855a0f49be9cbfbdcf3e715862" modify_external_catalogs: true
I20250623 08:04:23.105095 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet bf435a89f4f44a299b0ef5f4fc3891e6
I20250623 08:04:23.105628 32122 catalog_manager.cc:6652] ReplaceTablet: tablet bf435a89f4f44a299b0ef5f4fc3891e6 deleted and replaced by tablet ba26e7c2c2bd4deea68277dd01c1cbe0
I20250623 08:04:23.105937 32122 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_b7f62bf95f5843e8a1e9e0a08386bf06" } new_table_name: "default.Table_347f265adebd45bdb27956d1affc3e89" modify_external_catalogs: true
I20250623 08:04:23.107769 32124 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_5407838a4fc64b35bc677c8949d65491" } new_table_name: "default.Table_af33ad29cfd64f1d8fb5d135187e6ac4" modify_external_catalogs: true
I20250623 08:04:23.110020 32495 tablet_service.cc:1468] Processing CreateTablet for tablet d3a3c5a446184329b55fcc2cbf969831 (DEFAULT_TABLE table=default.Table_aeb7585a9aad4577b59b1b1164aa016e [id=cf9d65382d8b4fd98f18c628c2ccad23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.110172 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_1ef4fa3350714732a2ecec291c255ed6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.110790 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ef4fa3350714732a2ecec291c255ed6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.111183 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a3c5a446184329b55fcc2cbf969831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.111450 324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.69.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.60:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.69.60
--webserver_port=35145
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 07:50:47 UTC on 24a791456cd2
build id 6728
ASAN enabled
I20250623 08:04:23.113544 324 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 08:04:23.116153 324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250623 08:04:23.116596 343 log.cc:826] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: Log is configured to *not* fsync() on all Append() calls
I20250623 08:04:23.120718 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c7d30889c024c84b6585f007504139e with delete_type TABLET_DATA_DELETED (replaced by tablet b9d871507dcc46fc80f6e0ac739c6476) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.123724 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c7d30889c024c84b6585f007504139e with delete_type TABLET_DATA_DELETED (replaced by tablet b9d871507dcc46fc80f6e0ac739c6476) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.124393 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet b0fc9178845f4d68afcd54c430664c4c with delete_type TABLET_DATA_DELETED (replaced by tablet e0c97148429046dc88213ff70a605c99) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.124732 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c7d30889c024c84b6585f007504139e with delete_type TABLET_DATA_DELETED (replaced by tablet b9d871507dcc46fc80f6e0ac739c6476) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.125455 343 tablet_bootstrap.cc:492] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:23.125732 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet b0fc9178845f4d68afcd54c430664c4c with delete_type TABLET_DATA_DELETED (replaced by tablet e0c97148429046dc88213ff70a605c99) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.126083 344 tablet_replica.cc:331] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:23.126350 344 raft_consensus.cc:2241] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.126562 344 raft_consensus.cc:2270] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.125530 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet b0fc9178845f4d68afcd54c430664c4c with delete_type TABLET_DATA_DELETED (replaced by tablet e0c97148429046dc88213ff70a605c99) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.127099 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet bf435a89f4f44a299b0ef5f4fc3891e6 with delete_type TABLET_DATA_DELETED (replaced by tablet ba26e7c2c2bd4deea68277dd01c1cbe0) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.127337 344 ts_tablet_manager.cc:1905] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.128026 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet bf435a89f4f44a299b0ef5f4fc3891e6 with delete_type TABLET_DATA_DELETED (replaced by tablet ba26e7c2c2bd4deea68277dd01c1cbe0) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:23.128526 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 5c7d30889c024c84b6585f007504139e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c7d30889c024c84b6585f007504139e already in progress: creating tablet
W20250623 08:04:23.129071 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 5c7d30889c024c84b6585f007504139e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c7d30889c024c84b6585f007504139e already in progress: creating tablet
W20250623 08:04:23.129276 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b0fc9178845f4d68afcd54c430664c4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0fc9178845f4d68afcd54c430664c4c already in progress: creating tablet
W20250623 08:04:23.129465 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 5c7d30889c024c84b6585f007504139e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c7d30889c024c84b6585f007504139e already in progress: creating tablet
W20250623 08:04:23.129721 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet b0fc9178845f4d68afcd54c430664c4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0fc9178845f4d68afcd54c430664c4c already in progress: creating tablet
W20250623 08:04:23.129871 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet bf435a89f4f44a299b0ef5f4fc3891e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf435a89f4f44a299b0ef5f4fc3891e6 already in progress: creating tablet
I20250623 08:04:23.131969 339 raft_consensus.cc:357] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.132345 339 raft_consensus.cc:383] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.132452 339 raft_consensus.cc:738] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.133231 339 consensus_queue.cc:260] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.135102 344 ts_tablet_manager.cc:1918] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.135465 344 log.cc:1199] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b0fc9178845f4d68afcd54c430664c4c
W20250623 08:04:23.135637 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet bf435a89f4f44a299b0ef5f4fc3891e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf435a89f4f44a299b0ef5f4fc3891e6 already in progress: creating tablet
I20250623 08:04:23.136152 344 ts_tablet_manager.cc:1939] T b0fc9178845f4d68afcd54c430664c4c P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:23.137553 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet bf435a89f4f44a299b0ef5f4fc3891e6 with delete_type TABLET_DATA_DELETED (replaced by tablet ba26e7c2c2bd4deea68277dd01c1cbe0) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.137662 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b0fc9178845f4d68afcd54c430664c4c (table default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]) successfully deleted
I20250623 08:04:23.138283 343 ts_tablet_manager.cc:1397] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.158s user 0.010s sys 0.000s
I20250623 08:04:23.144541 344 tablet_replica.cc:331] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:23.145140 344 raft_consensus.cc:2241] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.145622 339 ts_tablet_manager.cc:1428] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.106s user 0.011s sys 0.008s
I20250623 08:04:23.145706 344 raft_consensus.cc:2270] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.146148 339 tablet_bootstrap.cc:492] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.147066 32428 heartbeater.cc:499] Master 127.31.69.61:33313 was elected leader, sending a full tablet report...
I20250623 08:04:23.148800 339 tablet_bootstrap.cc:654] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.148959 344 ts_tablet_manager.cc:1905] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:23.149173 361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:23.150614 342 tablet_bootstrap.cc:492] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.150947 342 ts_tablet_manager.cc:1397] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.027s
I20250623 08:04:23.152007 342 raft_consensus.cc:357] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:23.152197 363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:23.152382 342 raft_consensus.cc:383] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.152495 342 raft_consensus.cc:738] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.152868 342 consensus_queue.cc:260] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.157896 32121 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_13b0e15543044cd58b65937ca08d8df4" } modify_external_catalogs: true
I20250623 08:04:23.158104 32121 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_13b0e15543044cd58b65937ca08d8df4" } modify_external_catalogs: true
I20250623 08:04:23.158147 343 raft_consensus.cc:357] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.158563 343 raft_consensus.cc:383] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.158733 343 raft_consensus.cc:738] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.159142 32128 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_aeb7585a9aad4577b59b1b1164aa016e" } modify_external_catalogs: true
I20250623 08:04:23.159315 32128 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_aeb7585a9aad4577b59b1b1164aa016e" } modify_external_catalogs: true
I20250623 08:04:23.159646 343 consensus_queue.cc:260] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.155781 342 ts_tablet_manager.cc:1428] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:23.159993 32123 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_fac8d606d7454d3184af3a8d4c20ccc4" } modify_external_catalogs: true
I20250623 08:04:23.160146 32123 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_fac8d606d7454d3184af3a8d4c20ccc4" } modify_external_catalogs: true
I20250623 08:04:23.158638 32125 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_097551427ef74c7691bb08662c2c088c" } modify_external_catalogs: true
I20250623 08:04:23.162338 32125 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_097551427ef74c7691bb08662c2c088c" } modify_external_catalogs: true
I20250623 08:04:23.162642 342 tablet_bootstrap.cc:492] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.164822 342 tablet_bootstrap.cc:654] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.168229 339 tablet_bootstrap.cc:492] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.168448 339 ts_tablet_manager.cc:1397] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.007s
W20250623 08:04:23.168541 368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:23.169442 344 ts_tablet_manager.cc:1918] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.169410 339 raft_consensus.cc:357] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.169653 344 log.cc:1199] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/bf435a89f4f44a299b0ef5f4fc3891e6
I20250623 08:04:23.169672 339 raft_consensus.cc:383] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.169750 339 raft_consensus.cc:738] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.170033 339 consensus_queue.cc:260] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.170586 339 ts_tablet_manager.cc:1428] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:23.170994 339 tablet_bootstrap.cc:492] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.173096 344 ts_tablet_manager.cc:1939] T bf435a89f4f44a299b0ef5f4fc3891e6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:23.173106 32127 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_0beadbd550664534bbf90d51ca7a4b0b" } modify_external_catalogs: true
I20250623 08:04:23.173280 32127 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_0beadbd550664534bbf90d51ca7a4b0b" } modify_external_catalogs: true
I20250623 08:04:23.175160 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet bf435a89f4f44a299b0ef5f4fc3891e6 (table default.Table_097551427ef74c7691bb08662c2c088c [id=5630a19db2d046d6b7e94f8d724a13ab]) successfully deleted
I20250623 08:04:23.176087 32495 tablet_service.cc:1468] Processing CreateTablet for tablet 0f459b85abba44d288b0218c187f0098 (DEFAULT_TABLE table=default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.176093 32497 tablet_service.cc:1468] Processing CreateTablet for tablet a3bd7fa007544af69b3628911dca3a80 (DEFAULT_TABLE table=default.Table_0beadbd550664534bbf90d51ca7a4b0b [id=967a9b35621a401681cb78fb66402525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.176394 32130 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c7d66b3414784129a5fd5941860666cd (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]). Expected version 1 got 0
I20250623 08:04:23.176487 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f459b85abba44d288b0218c187f0098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.176679 32703 heartbeater.cc:499] Master 127.31.69.61:33313 was elected leader, sending a full tablet report...
I20250623 08:04:23.176716 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3bd7fa007544af69b3628911dca3a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.177412 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 0f459b85abba44d288b0218c187f0098 (DEFAULT_TABLE table=default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.177817 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f459b85abba44d288b0218c187f0098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.178504 32632 tablet_service.cc:1468] Processing CreateTablet for tablet a3bd7fa007544af69b3628911dca3a80 (DEFAULT_TABLE table=default.Table_0beadbd550664534bbf90d51ca7a4b0b [id=967a9b35621a401681cb78fb66402525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.178930 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3bd7fa007544af69b3628911dca3a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.179612 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 0f459b85abba44d288b0218c187f0098 (DEFAULT_TABLE table=default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.179975 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f459b85abba44d288b0218c187f0098. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.183053 32358 tablet_service.cc:1468] Processing CreateTablet for tablet a3bd7fa007544af69b3628911dca3a80 (DEFAULT_TABLE table=default.Table_0beadbd550664534bbf90d51ca7a4b0b [id=967a9b35621a401681cb78fb66402525]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.183398 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3bd7fa007544af69b3628911dca3a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.185176 342 tablet_bootstrap.cc:492] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.185365 342 ts_tablet_manager.cc:1397] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20250623 08:04:23.186473 342 raft_consensus.cc:357] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.186702 342 raft_consensus.cc:383] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.186851 342 raft_consensus.cc:738] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.187150 342 consensus_queue.cc:260] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.188056 342 ts_tablet_manager.cc:1428] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:23.188436 342 tablet_bootstrap.cc:492] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.190259 342 tablet_bootstrap.cc:654] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.194093 343 ts_tablet_manager.cc:1428] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.055s user 0.008s sys 0.005s
I20250623 08:04:23.194238 339 tablet_bootstrap.cc:654] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.194684 324 server_base.cc:1048] running on GCE node
I20250623 08:04:23.196135 324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 08:04:23.200845 32127 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet a3bd7fa007544af69b3628911dca3a80
I20250623 08:04:23.202525 343 tablet_bootstrap.cc:492] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:23.202841 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet a3bd7fa007544af69b3628911dca3a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.203302 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet a3bd7fa007544af69b3628911dca3a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
W20250623 08:04:23.203675 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet a3bd7fa007544af69b3628911dca3a80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3bd7fa007544af69b3628911dca3a80 already in progress: creating tablet
I20250623 08:04:23.203817 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet a3bd7fa007544af69b3628911dca3a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:23.204449 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet a3bd7fa007544af69b3628911dca3a80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3bd7fa007544af69b3628911dca3a80 already in progress: creating tablet
W20250623 08:04:23.204856 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet a3bd7fa007544af69b3628911dca3a80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3bd7fa007544af69b3628911dca3a80 already in progress: creating tablet
I20250623 08:04:23.205044 343 tablet_bootstrap.cc:654] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.205991 324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250623 08:04:23.206166 342 tablet_bootstrap.cc:492] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.206385 342 ts_tablet_manager.cc:1397] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20250623 08:04:23.206537 339 tablet_bootstrap.cc:492] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.206712 339 ts_tablet_manager.cc:1397] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.007s
I20250623 08:04:23.207242 342 raft_consensus.cc:357] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.207481 324 hybrid_clock.cc:648] HybridClock initialized: now 1750665863207305 us; error 126 us; skew 500 ppm
I20250623 08:04:23.207506 342 raft_consensus.cc:383] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.207592 342 raft_consensus.cc:738] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.207536 339 raft_consensus.cc:357] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.207700 339 raft_consensus.cc:383] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.207816 339 raft_consensus.cc:738] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.207911 342 consensus_queue.cc:260] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.208065 324 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 08:04:23.208098 339 consensus_queue.cc:260] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.208631 339 ts_tablet_manager.cc:1428] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:23.208999 339 tablet_bootstrap.cc:492] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.210865 342 ts_tablet_manager.cc:1428] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:23.211557 339 tablet_bootstrap.cc:654] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.217242 342 tablet_bootstrap.cc:492] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.220660 342 tablet_bootstrap.cc:654] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.221058 339 tablet_bootstrap.cc:492] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.221277 339 ts_tablet_manager.cc:1397] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20250623 08:04:23.222237 339 raft_consensus.cc:357] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.222625 339 raft_consensus.cc:383] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.222795 339 raft_consensus.cc:738] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.223052 343 tablet_bootstrap.cc:492] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:23.223294 343 ts_tablet_manager.cc:1397] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20250623 08:04:23.223194 339 consensus_queue.cc:260] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.224337 339 ts_tablet_manager.cc:1428] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:23.224789 339 tablet_bootstrap.cc:492] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.225967 32129 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c7d66b3414784129a5fd5941860666cd (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]). Expected version 1 got 0
I20250623 08:04:23.227936 339 tablet_bootstrap.cc:654] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.229668 343 raft_consensus.cc:357] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.229970 32127 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c7d66b3414784129a5fd5941860666cd (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]). Expected version 1 got 0
I20250623 08:04:23.230664 32121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 3bf5243f44c043a0a1e9bc441fc91e70
I20250623 08:04:23.230839 32125 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ba26e7c2c2bd4deea68277dd01c1cbe0
I20250623 08:04:23.231138 32125 master_service.cc:948] ReplaceTablet: received request to replace tablet 5c7d30889c024c84b6585f007504139e from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.231245 343 raft_consensus.cc:383] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.231348 343 raft_consensus.cc:738] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:23.231628 32124 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99ea7850738743deb7afa87ea2e40daa consensus state has no leader
I20250623 08:04:23.232012 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bf5243f44c043a0a1e9bc441fc91e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.232115 32123 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 9b5a56fe461641bea2bb269bf0564415
I20250623 08:04:23.232293 344 tablet_replica.cc:331] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:23.232467 344 raft_consensus.cc:2241] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.232666 344 raft_consensus.cc:2270] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.232969 343 consensus_queue.cc:260] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.233426 344 ts_tablet_manager.cc:1905] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.234094 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet d3a3c5a446184329b55fcc2cbf969831
I20250623 08:04:23.234441 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet bf435a89f4f44a299b0ef5f4fc3891e6 with delete_type TABLET_DATA_DELETED (replaced by tablet ba26e7c2c2bd4deea68277dd01c1cbe0) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.234925 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet b0fc9178845f4d68afcd54c430664c4c with delete_type TABLET_DATA_DELETED (replaced by tablet e0c97148429046dc88213ff70a605c99) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.235283 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bf5243f44c043a0a1e9bc441fc91e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.235502 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b5a56fe461641bea2bb269bf0564415 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.235715 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bf5243f44c043a0a1e9bc441fc91e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:23.236251 32122 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6e29ab07df249748092a2f50a8c0273 consensus state has no leader
I20250623 08:04:23.236687 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a3c5a446184329b55fcc2cbf969831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.236974 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a3c5a446184329b55fcc2cbf969831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.236969 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a3c5a446184329b55fcc2cbf969831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
W20250623 08:04:23.230659 32126 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75dde3beac4e47879c3f5e16151a4675 consensus state has no leader
I20250623 08:04:23.237459 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b5a56fe461641bea2bb269bf0564415 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.237581 343 ts_tablet_manager.cc:1428] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
W20250623 08:04:23.239326 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 3bf5243f44c043a0a1e9bc441fc91e70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bf5243f44c043a0a1e9bc441fc91e70 already in progress: creating tablet
W20250623 08:04:23.239634 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d3a3c5a446184329b55fcc2cbf969831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a3c5a446184329b55fcc2cbf969831 already in progress: creating tablet
W20250623 08:04:23.239863 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 3bf5243f44c043a0a1e9bc441fc91e70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bf5243f44c043a0a1e9bc441fc91e70 already in progress: creating tablet
W20250623 08:04:23.240068 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 9b5a56fe461641bea2bb269bf0564415 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b5a56fe461641bea2bb269bf0564415 already in progress: creating tablet
I20250623 08:04:23.240079 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet bf435a89f4f44a299b0ef5f4fc3891e6 with delete_type TABLET_DATA_DELETED (replaced by tablet ba26e7c2c2bd4deea68277dd01c1cbe0) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.240283 32356 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c7d30889c024c84b6585f007504139e with delete_type TABLET_DATA_DELETED (replaced by tablet b9d871507dcc46fc80f6e0ac739c6476) from {username='slave'} at 127.0.0.1:35526
W20250623 08:04:23.240319 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet d3a3c5a446184329b55fcc2cbf969831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a3c5a446184329b55fcc2cbf969831 already in progress: creating tablet
I20250623 08:04:23.240434 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet b0fc9178845f4d68afcd54c430664c4c with delete_type TABLET_DATA_DELETED (replaced by tablet e0c97148429046dc88213ff70a605c99) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.240581 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ee90c0361c491b94ca9ff06962de1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.240854 340 tablet_replica.cc:331] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:23.241164 344 ts_tablet_manager.cc:1918] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.241277 344 log.cc:1199] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/3bf5243f44c043a0a1e9bc441fc91e70
I20250623 08:04:23.241281 343 tablet_bootstrap.cc:492] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:23.241390 340 raft_consensus.cc:2241] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.241639 344 ts_tablet_manager.cc:1939] T 3bf5243f44c043a0a1e9bc441fc91e70 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:23.241803 340 raft_consensus.cc:2270] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.243084 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b5a56fe461641bea2bb269bf0564415 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.243643 340 ts_tablet_manager.cc:1905] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.245033 32128 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]). Expected version 1 got 0
I20250623 08:04:23.245297 32128 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 75dde3beac4e47879c3f5e16151a4675 (table default.Table_3f74ba855a0f49be9cbfbdcf3e715862 [id=26c5fff9eaea4788844a55e87ada7215]). Expected version 1 got 0
I20250623 08:04:23.246836 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c5e77c518e646e19e67c84a8f687801 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.248268 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ee90c0361c491b94ca9ff06962de1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.248507 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet bf435a89f4f44a299b0ef5f4fc3891e6 with delete_type TABLET_DATA_DELETED (replaced by tablet ba26e7c2c2bd4deea68277dd01c1cbe0) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.248724 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a3c5a446184329b55fcc2cbf969831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:23.248839 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9b5a56fe461641bea2bb269bf0564415 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b5a56fe461641bea2bb269bf0564415 already in progress: creating tablet
I20250623 08:04:23.248924 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bf5243f44c043a0a1e9bc441fc91e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:23.249063 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 3c5e77c518e646e19e67c84a8f687801 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c5e77c518e646e19e67c84a8f687801 already in progress: creating tablet
I20250623 08:04:23.249109 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet b0fc9178845f4d68afcd54c430664c4c with delete_type TABLET_DATA_DELETED (replaced by tablet e0c97148429046dc88213ff70a605c99) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.249230 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 3bf5243f44c043a0a1e9bc441fc91e70 (table default.Table_13b0e15543044cd58b65937ca08d8df4 [id=adb0245ebf874699887e167d5b46d1d2]) successfully deleted
W20250623 08:04:23.249418 32114 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet bf435a89f4f44a299b0ef5f4fc3891e6 because the tablet was not found. No further retry: Not found: Tablet not found: bf435a89f4f44a299b0ef5f4fc3891e6
I20250623 08:04:23.249559 341 tablet_replica.cc:331] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
W20250623 08:04:23.249612 32114 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet b0fc9178845f4d68afcd54c430664c4c because the tablet was not found. No further retry: Not found: Tablet not found: b0fc9178845f4d68afcd54c430664c4c
I20250623 08:04:23.249907 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c7d30889c024c84b6585f007504139e with delete_type TABLET_DATA_DELETED (replaced by tablet b9d871507dcc46fc80f6e0ac739c6476) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:23.250008 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet d3a3c5a446184329b55fcc2cbf969831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a3c5a446184329b55fcc2cbf969831 already in progress: creating tablet
W20250623 08:04:23.250187 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 9b5a56fe461641bea2bb269bf0564415 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b5a56fe461641bea2bb269bf0564415 already in progress: creating tablet
I20250623 08:04:23.250376 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c5e77c518e646e19e67c84a8f687801 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.250768 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c5e77c518e646e19e67c84a8f687801 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.251124 340 ts_tablet_manager.cc:1918] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.251142 341 raft_consensus.cc:2241] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.251271 340 log.cc:1199] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/bf435a89f4f44a299b0ef5f4fc3891e6
I20250623 08:04:23.251566 341 raft_consensus.cc:2270] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.251736 340 ts_tablet_manager.cc:1939] T bf435a89f4f44a299b0ef5f4fc3891e6 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:23.253994 344 tablet_replica.cc:331] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:23.254295 344 raft_consensus.cc:2241] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.254529 344 raft_consensus.cc:2270] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.254722 341 ts_tablet_manager.cc:1905] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.255486 32359 tablet_service.cc:1468] Processing CreateTablet for tablet abf018986c24473688c8e5eec724f8ee (DEFAULT_TABLE table=default.Table_1ef4fa3350714732a2ecec291c255ed6 [id=affccd38bcd54f73af82d0ceeba931f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.255631 32633 tablet_service.cc:1468] Processing CreateTablet for tablet abf018986c24473688c8e5eec724f8ee (DEFAULT_TABLE table=default.Table_1ef4fa3350714732a2ecec291c255ed6 [id=affccd38bcd54f73af82d0ceeba931f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.255899 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf018986c24473688c8e5eec724f8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.256078 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf018986c24473688c8e5eec724f8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.260401 339 tablet_bootstrap.cc:492] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.260550 339 ts_tablet_manager.cc:1397] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.005s
I20250623 08:04:23.261394 339 raft_consensus.cc:357] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.261570 339 raft_consensus.cc:383] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.261656 339 raft_consensus.cc:738] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.261864 339 consensus_queue.cc:260] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.262782 32497 tablet_service.cc:1468] Processing CreateTablet for tablet abf018986c24473688c8e5eec724f8ee (DEFAULT_TABLE table=default.Table_1ef4fa3350714732a2ecec291c255ed6 [id=affccd38bcd54f73af82d0ceeba931f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.262918 32358 tablet_service.cc:1468] Processing CreateTablet for tablet b1db513f73d24116af299da4d267aabf (DEFAULT_TABLE table=default.Table_ced228608de64c2892b15f6dbd5a2d4d [id=491fdbceb3d448d3904beb75d0881652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.263299 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf018986c24473688c8e5eec724f8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.263402 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1db513f73d24116af299da4d267aabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.263445 339 ts_tablet_manager.cc:1428] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250623 08:04:23.263759 339 tablet_bootstrap.cc:492] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.263768 32356 tablet_service.cc:1468] Processing CreateTablet for tablet e6eb8420d9c849a5a751ed38fdac53a4 (DEFAULT_TABLE table=default.Table_0e539ce89ea247e8b92fae1cc603ca88 [id=4e8632de2b4744ff9de198f82720a5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.265061 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6eb8420d9c849a5a751ed38fdac53a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:23.265367 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b6ee90c0361c491b94ca9ff06962de1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6ee90c0361c491b94ca9ff06962de1b already in progress: creating tablet
I20250623 08:04:23.265801 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet bf435a89f4f44a299b0ef5f4fc3891e6 (table default.Table_097551427ef74c7691bb08662c2c088c [id=5630a19db2d046d6b7e94f8d724a13ab]) successfully deleted
I20250623 08:04:23.268587 32499 tablet_service.cc:1468] Processing CreateTablet for tablet e6eb8420d9c849a5a751ed38fdac53a4 (DEFAULT_TABLE table=default.Table_0e539ce89ea247e8b92fae1cc603ca88 [id=4e8632de2b4744ff9de198f82720a5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.268723 341 ts_tablet_manager.cc:1918] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.268898 341 log.cc:1199] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/bf435a89f4f44a299b0ef5f4fc3891e6
I20250623 08:04:23.269047 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6eb8420d9c849a5a751ed38fdac53a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.269335 341 ts_tablet_manager.cc:1939] T bf435a89f4f44a299b0ef5f4fc3891e6 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:23.270328 32493 tablet_service.cc:1468] Processing CreateTablet for tablet b9d871507dcc46fc80f6e0ac739c6476 (DEFAULT_TABLE table=default.Table_d27db7e0fea34b5182353dee8ae23cac [id=816747bd5e004ae5964b7b885aa249fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.270872 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9d871507dcc46fc80f6e0ac739c6476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.271070 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 1e0c11ec768a43a48c6dc0f3dd974b71 (DEFAULT_TABLE table=default.Table_c2bc4709093c46c7a0db851d422b41a0 [id=bb849e86d7084959830c68207fb99061]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.271407 341 tablet_replica.cc:331] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:23.271502 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0c11ec768a43a48c6dc0f3dd974b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.271579 341 raft_consensus.cc:2241] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.271720 341 raft_consensus.cc:2270] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.274081 343 tablet_bootstrap.cc:654] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.275091 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 9b6f3fe1b10143aa9b54022546812498 (DEFAULT_TABLE table=default.Table_d2bb692b2c70474ea4b997cde4fe161a [id=33784d2b4a194d38b56b4881313dc03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.275527 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6f3fe1b10143aa9b54022546812498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.277738 32494 tablet_service.cc:1468] Processing CreateTablet for tablet ed5c81d9e56c4ee28546cedb9a579f31 (DEFAULT_TABLE table=default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.278276 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5c81d9e56c4ee28546cedb9a579f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.280561 319 raft_consensus.cc:491] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:23.280689 319 raft_consensus.cc:513] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.281288 319 leader_election.cc:290] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:23.281481 32496 tablet_service.cc:1468] Processing CreateTablet for tablet e0c97148429046dc88213ff70a605c99 (DEFAULT_TABLE table=default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.281795 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0c97148429046dc88213ff70a605c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.281988 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 7300f526d91f441bb164fe47f3a4f0ce (DEFAULT_TABLE table=default.Table_2ab537bbc9724fd69812910c17dca994 [id=cb2c6ba697ca494c8eb60b65624a0517]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.282027 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75dde3beac4e47879c3f5e16151a4675" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:23.282430 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7300f526d91f441bb164fe47f3a4f0ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:23.282810 32453 leader_election.cc:343] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:23.285619 32429 tablet.cc:2378] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 08:04:23.286473 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_465c441ac95145f1b498665ccedafe6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.287091 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_465c441ac95145f1b498665ccedafe6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.289944 32634 tablet_service.cc:1468] Processing CreateTablet for tablet b1db513f73d24116af299da4d267aabf (DEFAULT_TABLE table=default.Table_ced228608de64c2892b15f6dbd5a2d4d [id=491fdbceb3d448d3904beb75d0881652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.290325 340 tablet_replica.cc:331] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:23.290386 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1db513f73d24116af299da4d267aabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.290521 32635 tablet_service.cc:1468] Processing CreateTablet for tablet ed5c81d9e56c4ee28546cedb9a579f31 (DEFAULT_TABLE table=default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.290933 32629 tablet_service.cc:1468] Processing CreateTablet for tablet b9d871507dcc46fc80f6e0ac739c6476 (DEFAULT_TABLE table=default.Table_d27db7e0fea34b5182353dee8ae23cac [id=816747bd5e004ae5964b7b885aa249fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.291775 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 1e0c11ec768a43a48c6dc0f3dd974b71 (DEFAULT_TABLE table=default.Table_c2bc4709093c46c7a0db851d422b41a0 [id=bb849e86d7084959830c68207fb99061]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.292008 32125 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c2bc4709093c46c7a0db851d422b41a0" } new_table_name: "default.Table_a564e263b16445d28147f335d686b132" modify_external_catalogs: true
I20250623 08:04:23.292181 32495 tablet_service.cc:1468] Processing CreateTablet for tablet b1db513f73d24116af299da4d267aabf (DEFAULT_TABLE table=default.Table_ced228608de64c2892b15f6dbd5a2d4d [id=491fdbceb3d448d3904beb75d0881652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.292600 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1db513f73d24116af299da4d267aabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.294008 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_40311f16d5ff467e9044de06e628afe9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.294540 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40311f16d5ff467e9044de06e628afe9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.295454 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet bf435a89f4f44a299b0ef5f4fc3891e6 (table default.Table_097551427ef74c7691bb08662c2c088c [id=5630a19db2d046d6b7e94f8d724a13ab]) successfully deleted
W20250623 08:04:23.296703 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d3a3c5a446184329b55fcc2cbf969831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a3c5a446184329b55fcc2cbf969831 already in progress: creating tablet
W20250623 08:04:23.297049 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 3bf5243f44c043a0a1e9bc441fc91e70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bf5243f44c043a0a1e9bc441fc91e70 already in progress: creating tablet
I20250623 08:04:23.297030 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 9b6f3fe1b10143aa9b54022546812498 (DEFAULT_TABLE table=default.Table_d2bb692b2c70474ea4b997cde4fe161a [id=33784d2b4a194d38b56b4881313dc03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.297683 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 1e0c11ec768a43a48c6dc0f3dd974b71 (DEFAULT_TABLE table=default.Table_c2bc4709093c46c7a0db851d422b41a0 [id=bb849e86d7084959830c68207fb99061]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.298015 32632 tablet_service.cc:1468] Processing CreateTablet for tablet e0c97148429046dc88213ff70a605c99 (DEFAULT_TABLE table=default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.298249 32630 tablet_service.cc:1468] Processing CreateTablet for tablet e6eb8420d9c849a5a751ed38fdac53a4 (DEFAULT_TABLE table=default.Table_0e539ce89ea247e8b92fae1cc603ca88 [id=4e8632de2b4744ff9de198f82720a5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.299337 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6f3fe1b10143aa9b54022546812498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.300122 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 125784661a7b4e4ba6a710b950e004f8 (DEFAULT_TABLE table=default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.300644 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_ced228608de64c2892b15f6dbd5a2d4d" } new_table_name: "default.Table_cd5fd778975c47eb8aa915cb59bd588c" modify_external_catalogs: true
I20250623 08:04:23.300892 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 7300f526d91f441bb164fe47f3a4f0ce (DEFAULT_TABLE table=default.Table_2ab537bbc9724fd69812910c17dca994 [id=cb2c6ba697ca494c8eb60b65624a0517]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.301649 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0c11ec768a43a48c6dc0f3dd974b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.306840 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6eb8420d9c849a5a751ed38fdac53a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.307762 32129 master_service.cc:948] ReplaceTablet: received request to replace tablet c7d66b3414784129a5fd5941860666cd from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.309614 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_403b5a6a8805486f90514f62555d80a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.311539 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_403b5a6a8805486f90514f62555d80a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.312372 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 9b6f3fe1b10143aa9b54022546812498 (DEFAULT_TABLE table=default.Table_d2bb692b2c70474ea4b997cde4fe161a [id=33784d2b4a194d38b56b4881313dc03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.312572 32360 tablet_service.cc:1468] Processing CreateTablet for tablet ed5c81d9e56c4ee28546cedb9a579f31 (DEFAULT_TABLE table=default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.313336 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0c97148429046dc88213ff70a605c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.314429 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5c81d9e56c4ee28546cedb9a579f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.315609 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9d871507dcc46fc80f6e0ac739c6476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.316057 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125784661a7b4e4ba6a710b950e004f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.316593 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7300f526d91f441bb164fe47f3a4f0ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:23.320326 32125 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e0c11ec768a43a48c6dc0f3dd974b71 consensus state has no leader
I20250623 08:04:23.289700 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75dde3beac4e47879c3f5e16151a4675" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:23.323210 344 ts_tablet_manager.cc:1905] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.330139 344 ts_tablet_manager.cc:1918] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.330331 344 log.cc:1199] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/3c5e77c518e646e19e67c84a8f687801
I20250623 08:04:23.330699 344 ts_tablet_manager.cc:1939] T 3c5e77c518e646e19e67c84a8f687801 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:23.330658 32122 master_service.cc:948] ReplaceTablet: received request to replace tablet b9d871507dcc46fc80f6e0ac739c6476 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.331990 32128 master_service.cc:948] ReplaceTablet: received request to replace tablet 3c5e77c518e646e19e67c84a8f687801 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.334865 32128 master_service.cc:948] ReplaceTablet: received request to replace tablet 0f459b85abba44d288b0218c187f0098 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.336249 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_d134057b3f2b420290f1b7f3aa73414a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.336791 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d134057b3f2b420290f1b7f3aa73414a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.337544 339 tablet_bootstrap.cc:654] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:23.337929 32453 leader_election.cc:343] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:23.338063 32453 leader_election.cc:304] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:23.338558 319 raft_consensus.cc:2747] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:23.338721 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e0c11ec768a43a48c6dc0f3dd974b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.339720 32357 tablet_service.cc:1468] Processing CreateTablet for tablet e0c97148429046dc88213ff70a605c99 (DEFAULT_TABLE table=default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:23.341045 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1db513f73d24116af299da4d267aabf consensus state has no leader
I20250623 08:04:23.341030 32355 tablet_service.cc:1468] Processing CreateTablet for tablet b9d871507dcc46fc80f6e0ac739c6476 (DEFAULT_TABLE table=default.Table_d27db7e0fea34b5182353dee8ae23cac [id=816747bd5e004ae5964b7b885aa249fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.341888 32129 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet c7d66b3414784129a5fd5941860666cd
I20250623 08:04:23.342136 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed5c81d9e56c4ee28546cedb9a579f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.342401 32129 catalog_manager.cc:6652] ReplaceTablet: tablet c7d66b3414784129a5fd5941860666cd deleted and replaced by tablet a46c0a89fc014054a75ed198e824b5b6
I20250623 08:04:23.343415 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 125784661a7b4e4ba6a710b950e004f8 (DEFAULT_TABLE table=default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.343816 342 tablet_bootstrap.cc:492] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
W20250623 08:04:23.343777 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 5c7d30889c024c84b6585f007504139e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c7d30889c024c84b6585f007504139e already in progress: creating tablet
I20250623 08:04:23.343993 342 ts_tablet_manager.cc:1397] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.127s user 0.015s sys 0.021s
I20250623 08:04:23.344780 342 raft_consensus.cc:357] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.344991 342 raft_consensus.cc:383] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.345091 342 raft_consensus.cc:738] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.345271 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 3c5e77c518e646e19e67c84a8f687801 (table default.Table_25af89afee014e4bbecdb22a93780edc [id=9a792e7fecc84b6dab8eeced687eed5d]) successfully deleted
I20250623 08:04:23.345336 342 consensus_queue.cc:260] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.345870 342 ts_tablet_manager.cc:1428] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:23.346065 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet c7d66b3414784129a5fd5941860666cd with delete_type TABLET_DATA_DELETED (replaced by tablet a46c0a89fc014054a75ed198e824b5b6) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.346253 342 tablet_bootstrap.cc:492] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.348059 342 tablet_bootstrap.cc:654] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.348136 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet c7d66b3414784129a5fd5941860666cd with delete_type TABLET_DATA_DELETED (replaced by tablet a46c0a89fc014054a75ed198e824b5b6) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.348389 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6f3fe1b10143aa9b54022546812498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.348789 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet c7d66b3414784129a5fd5941860666cd with delete_type TABLET_DATA_DELETED (replaced by tablet a46c0a89fc014054a75ed198e824b5b6) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.349087 344 tablet_replica.cc:331] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:23.349267 344 raft_consensus.cc:2241] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.349411 324 webserver.cc:469] Webserver started at http://127.31.69.60:35145/ using document root <none> and password file <none>
I20250623 08:04:23.349457 344 raft_consensus.cc:2270] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.350190 344 ts_tablet_manager.cc:1905] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.351323 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 125784661a7b4e4ba6a710b950e004f8 (DEFAULT_TABLE table=default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.343127 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 7300f526d91f441bb164fe47f3a4f0ce (DEFAULT_TABLE table=default.Table_2ab537bbc9724fd69812910c17dca994 [id=cb2c6ba697ca494c8eb60b65624a0517]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.351775 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125784661a7b4e4ba6a710b950e004f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.354516 324 fs_manager.cc:362] Metadata directory not provided
I20250623 08:04:23.354688 324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 08:04:23.357065 344 ts_tablet_manager.cc:1918] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.357257 344 log.cc:1199] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/c7d66b3414784129a5fd5941860666cd
I20250623 08:04:23.357656 344 ts_tablet_manager.cc:1939] T c7d66b3414784129a5fd5941860666cd P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:23.357865 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0c97148429046dc88213ff70a605c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.358918 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet c7d66b3414784129a5fd5941860666cd (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]) successfully deleted
I20250623 08:04:23.359604 340 raft_consensus.cc:2241] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.361621 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c7d30889c024c84b6585f007504139e with delete_type TABLET_DATA_DELETED (replaced by tablet b9d871507dcc46fc80f6e0ac739c6476) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.361876 344 tablet_replica.cc:331] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:23.362069 344 raft_consensus.cc:2241] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.362231 344 raft_consensus.cc:2270] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.362787 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_c3e871b94f3944d6b2b75626c3af94ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.363085 344 ts_tablet_manager.cc:1905] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:23.363361 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3e871b94f3944d6b2b75626c3af94ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.365952 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_8e277513586142b9bfbe562aef2b8f33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.366132 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_47264042125847ac997923572c293a3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.366477 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e277513586142b9bfbe562aef2b8f33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:23.366778 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47264042125847ac997923572c293a3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.369961 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_61dd3355d3b74f0aaf0d391f30fcf754"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.370460 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61dd3355d3b74f0aaf0d391f30fcf754 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.372412 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_4ea46914693645a7a1db29644cecdb84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.372927 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ea46914693645a7a1db29644cecdb84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.375466 343 tablet_bootstrap.cc:492] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:23.375635 343 ts_tablet_manager.cc:1397] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.134s user 0.006s sys 0.024s
I20250623 08:04:23.376559 343 raft_consensus.cc:357] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.376750 343 raft_consensus.cc:383] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.376883 343 raft_consensus.cc:738] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.378376 341 ts_tablet_manager.cc:1905] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.384009 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_7a7bf77e25a045cf863d9e8b46b15ee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.384658 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a7bf77e25a045cf863d9e8b46b15ee7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.393054 343 consensus_queue.cc:260] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.393698 343 ts_tablet_manager.cc:1428] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250623 08:04:23.393988 343 tablet_bootstrap.cc:492] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:23.397172 344 ts_tablet_manager.cc:1918] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.397364 344 log.cc:1199] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/5c7d30889c024c84b6585f007504139e
I20250623 08:04:23.397856 344 ts_tablet_manager.cc:1939] T 5c7d30889c024c84b6585f007504139e P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:23.399140 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 5c7d30889c024c84b6585f007504139e (table default.Table_d27db7e0fea34b5182353dee8ae23cac [id=816747bd5e004ae5964b7b885aa249fb]) successfully deleted
I20250623 08:04:23.405368 343 tablet_bootstrap.cc:654] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.408007 324 fs_manager.cc:714] Time spent opening directory manager: real 0.050s user 0.005s sys 0.001s
I20250623 08:04:23.408124 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9d871507dcc46fc80f6e0ac739c6476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.409608 32126 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d2bb692b2c70474ea4b997cde4fe161a" } modify_external_catalogs: true
I20250623 08:04:23.409807 32126 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d2bb692b2c70474ea4b997cde4fe161a" } modify_external_catalogs: true
I20250623 08:04:23.410113 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 0f459b85abba44d288b0218c187f0098
I20250623 08:04:23.410569 32128 catalog_manager.cc:6652] ReplaceTablet: tablet 0f459b85abba44d288b0218c187f0098 deleted and replaced by tablet c1596e80225041099fb513ce477eae33
I20250623 08:04:23.411067 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet b9d871507dcc46fc80f6e0ac739c6476
I20250623 08:04:23.411468 32122 catalog_manager.cc:6652] ReplaceTablet: tablet b9d871507dcc46fc80f6e0ac739c6476 deleted and replaced by tablet ca99c3b0cfec4aa481b620668b20947e
I20250623 08:04:23.411912 340 raft_consensus.cc:2270] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.414039 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet b9d871507dcc46fc80f6e0ac739c6476 with delete_type TABLET_DATA_DELETED (replaced by tablet ca99c3b0cfec4aa481b620668b20947e) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.414317 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f459b85abba44d288b0218c187f0098 with delete_type TABLET_DATA_DELETED (replaced by tablet c1596e80225041099fb513ce477eae33) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.414330 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet b9d871507dcc46fc80f6e0ac739c6476 with delete_type TABLET_DATA_DELETED (replaced by tablet ca99c3b0cfec4aa481b620668b20947e) from {username='slave'} at 127.0.0.1:55518
W20250623 08:04:23.415023 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet b9d871507dcc46fc80f6e0ac739c6476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9d871507dcc46fc80f6e0ac739c6476 already in progress: creating tablet
I20250623 08:04:23.418381 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f459b85abba44d288b0218c187f0098 with delete_type TABLET_DATA_DELETED (replaced by tablet c1596e80225041099fb513ce477eae33) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.418723 340 ts_tablet_manager.cc:1905] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:23.418970 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 0f459b85abba44d288b0218c187f0098 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f459b85abba44d288b0218c187f0098 already in progress: creating tablet
I20250623 08:04:23.424021 384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:23.432071 32126 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 9b6f3fe1b10143aa9b54022546812498
I20250623 08:04:23.433249 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b6f3fe1b10143aa9b54022546812498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
W20250623 08:04:23.433918 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 9b6f3fe1b10143aa9b54022546812498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b6f3fe1b10143aa9b54022546812498 already in progress: creating tablet
I20250623 08:04:23.435129 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 125784661a7b4e4ba6a710b950e004f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.437110 32130 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8e277513586142b9bfbe562aef2b8f33" } modify_external_catalogs: true
I20250623 08:04:23.437311 32130 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8e277513586142b9bfbe562aef2b8f33" } modify_external_catalogs: true
I20250623 08:04:23.437779 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7300f526d91f441bb164fe47f3a4f0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.440850 32121 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_7a7bf77e25a045cf863d9e8b46b15ee7" } new_table_name: "default.Table_be622804eb114901b245383926c3afec" modify_external_catalogs: true
I20250623 08:04:23.442415 32129 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_af33ad29cfd64f1d8fb5d135187e6ac4" } new_table_name: "default.Table_787076d19b96478889d932e380a68e59" modify_external_catalogs: true
I20250623 08:04:23.443876 32124 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_61dd3355d3b74f0aaf0d391f30fcf754" } modify_external_catalogs: true
I20250623 08:04:23.444098 32124 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_61dd3355d3b74f0aaf0d391f30fcf754" } modify_external_catalogs: true
I20250623 08:04:23.445428 32123 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_465c441ac95145f1b498665ccedafe6c" } modify_external_catalogs: true
I20250623 08:04:23.445639 32123 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_465c441ac95145f1b498665ccedafe6c" } modify_external_catalogs: true
I20250623 08:04:23.450677 32126 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d27db7e0fea34b5182353dee8ae23cac" } modify_external_catalogs: true
I20250623 08:04:23.450924 32126 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d27db7e0fea34b5182353dee8ae23cac" } modify_external_catalogs: true
I20250623 08:04:23.452494 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_403b5a6a8805486f90514f62555d80a0" } new_table_name: "default.Table_c5dbc5b105b8416c90d3792d29568b83" modify_external_catalogs: true
I20250623 08:04:23.454802 32122 master_service.cc:948] ReplaceTablet: received request to replace tablet ed5c81d9e56c4ee28546cedb9a579f31 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.456008 32130 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 0db2343566504d529fd76c8a3425e335
I20250623 08:04:23.461199 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b6f3fe1b10143aa9b54022546812498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.464835 32493 tablet_service.cc:1468] Processing CreateTablet for tablet e90151394fb842498f4e780acab1f1aa (DEFAULT_TABLE table=default.Table_40311f16d5ff467e9044de06e628afe9 [id=a1d4326f22fa44ba82cb376970f55ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.465320 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90151394fb842498f4e780acab1f1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.465389 32500 tablet_service.cc:1468] Processing CreateTablet for tablet a46c0a89fc014054a75ed198e824b5b6 (DEFAULT_TABLE table=default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.465889 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46c0a89fc014054a75ed198e824b5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.468152 32497 tablet_service.cc:1468] Processing CreateTablet for tablet c7385d86316f4a3ab68a77d18080ec20 (DEFAULT_TABLE table=default.Table_465c441ac95145f1b498665ccedafe6c [id=24b0c7dcb97e4105b8edf674a0a90e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.468536 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7385d86316f4a3ab68a77d18080ec20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.472124 324 fs_manager.cc:730] Time spent opening block manager: real 0.058s user 0.006s sys 0.000s
W20250623 08:04:23.472292 32121 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d67a8f1f570741ebb886c4d136ca40d2 has no consensus state
W20250623 08:04:23.472316 32129 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99ea7850738743deb7afa87ea2e40daa consensus state has no leader
I20250623 08:04:23.472360 324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data,/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
uuid: "6644af0f4b2a4b998282b021ec1dc678"
format_stamp: "Formatted at 2025-06-23 08:04:19 on dist-test-slave-bxhn"
I20250623 08:04:23.473417 324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 08:04:23.493971 32356 tablet_service.cc:1468] Processing CreateTablet for tablet a46c0a89fc014054a75ed198e824b5b6 (DEFAULT_TABLE table=default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.494498 32352 tablet_service.cc:1468] Processing CreateTablet for tablet c7385d86316f4a3ab68a77d18080ec20 (DEFAULT_TABLE table=default.Table_465c441ac95145f1b498665ccedafe6c [id=24b0c7dcb97e4105b8edf674a0a90e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.496052 32124 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a0a8ff15c2ac4f00bcbc5bac054c3d83
I20250623 08:04:23.497478 32123 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet c7385d86316f4a3ab68a77d18080ec20
I20250623 08:04:23.497596 32126 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ca99c3b0cfec4aa481b620668b20947e
W20250623 08:04:23.497948 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35428260e05e447ebf88ea41e63cdd02 has no consensus state
I20250623 08:04:23.498258 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet ed5c81d9e56c4ee28546cedb9a579f31
I20250623 08:04:23.498651 32122 catalog_manager.cc:6652] ReplaceTablet: tablet ed5c81d9e56c4ee28546cedb9a579f31 deleted and replaced by tablet 04a3ae58d26142d58c4df4f52d7e3e2c
I20250623 08:04:23.499467 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet c7385d86316f4a3ab68a77d18080ec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.500572 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet ed5c81d9e56c4ee28546cedb9a579f31 with delete_type TABLET_DATA_DELETED (replaced by tablet 04a3ae58d26142d58c4df4f52d7e3e2c) from {username='slave'} at 127.0.0.1:55518
W20250623 08:04:23.501221 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet ed5c81d9e56c4ee28546cedb9a579f31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed5c81d9e56c4ee28546cedb9a579f31 already in progress: creating tablet
I20250623 08:04:23.501545 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet c7385d86316f4a3ab68a77d18080ec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
W20250623 08:04:23.502058 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet c7385d86316f4a3ab68a77d18080ec20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7385d86316f4a3ab68a77d18080ec20 already in progress: creating tablet
I20250623 08:04:23.503832 341 ts_tablet_manager.cc:1918] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.504822 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_13b2a4a5337e4aeab1c2578c2cee28a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.505424 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13b2a4a5337e4aeab1c2578c2cee28a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.506453 32358 tablet_service.cc:1468] Processing CreateTablet for tablet e90151394fb842498f4e780acab1f1aa (DEFAULT_TABLE table=default.Table_40311f16d5ff467e9044de06e628afe9 [id=a1d4326f22fa44ba82cb376970f55ddc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:23.507362 32704 tablet.cc:2378] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 08:04:23.507737 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46c0a89fc014054a75ed198e824b5b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.507659 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet ed5c81d9e56c4ee28546cedb9a579f31 with delete_type TABLET_DATA_DELETED (replaced by tablet 04a3ae58d26142d58c4df4f52d7e3e2c) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.508677 341 log.cc:1199] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b0fc9178845f4d68afcd54c430664c4c
I20250623 08:04:23.511965 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7385d86316f4a3ab68a77d18080ec20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.517537 362 raft_consensus.cc:491] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:23.518088 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90151394fb842498f4e780acab1f1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.517789 362 raft_consensus.cc:513] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.519277 32636 tablet_service.cc:1468] Processing CreateTablet for tablet e90151394fb842498f4e780acab1f1aa (DEFAULT_TABLE table=default.Table_40311f16d5ff467e9044de06e628afe9 [id=a1d4326f22fa44ba82cb376970f55ddc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.520288 362 leader_election.cc:290] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:23.522964 32630 tablet_service.cc:1468] Processing CreateTablet for tablet c7385d86316f4a3ab68a77d18080ec20 (DEFAULT_TABLE table=default.Table_465c441ac95145f1b498665ccedafe6c [id=24b0c7dcb97e4105b8edf674a0a90e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.525404 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_2807052f51c24614b4068e8113db2abd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.525496 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_4ea46914693645a7a1db29644cecdb84" } new_table_name: "default.Table_0b402e459582438db7f3ad072b276ee0" modify_external_catalogs: true
W20250623 08:04:23.525996 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2807052f51c24614b4068e8113db2abd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.526132 341 ts_tablet_manager.cc:1939] T b0fc9178845f4d68afcd54c430664c4c P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:23.531153 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b0fc9178845f4d68afcd54c430664c4c (table default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]) successfully deleted
I20250623 08:04:23.532276 32634 tablet_service.cc:1468] Processing CreateTablet for tablet a46c0a89fc014054a75ed198e824b5b6 (DEFAULT_TABLE table=default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.532570 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e90151394fb842498f4e780acab1f1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.534888 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet c7385d86316f4a3ab68a77d18080ec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.535113 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet ed5c81d9e56c4ee28546cedb9a579f31 with delete_type TABLET_DATA_DELETED (replaced by tablet 04a3ae58d26142d58c4df4f52d7e3e2c) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.535456 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7385d86316f4a3ab68a77d18080ec20. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:23.542868 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b has no consensus state
I20250623 08:04:23.544785 32122 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d134057b3f2b420290f1b7f3aa73414a" } new_table_name: "default.Table_815c7abc913347a688cc1772704c6715" modify_external_catalogs: true
I20250623 08:04:23.547041 341 tablet_replica.cc:331] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:23.547287 341 raft_consensus.cc:2241] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.547500 341 raft_consensus.cc:2270] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.548005 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46c0a89fc014054a75ed198e824b5b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:23.553442 32122 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ca4957928fc4630aa8d8ca6110d7598 has no consensus state
I20250623 08:04:23.554313 340 ts_tablet_manager.cc:1918] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.555501 340 log.cc:1199] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b0fc9178845f4d68afcd54c430664c4c
I20250623 08:04:23.555925 340 ts_tablet_manager.cc:1939] T b0fc9178845f4d68afcd54c430664c4c P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:23.559684 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet b0fc9178845f4d68afcd54c430664c4c (table default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]) successfully deleted
I20250623 08:04:23.559787 32633 tablet_service.cc:1468] Processing CreateTablet for tablet b6ad6cd629c4423ca997efb517cc975b (DEFAULT_TABLE table=default.Table_47264042125847ac997923572c293a3e [id=4214b1f0ca1744dc9b46d3f4d7618ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.560032 32626 tablet_service.cc:1468] Processing CreateTablet for tablet c1596e80225041099fb513ce477eae33 (DEFAULT_TABLE table=default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.560178 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6ad6cd629c4423ca997efb517cc975b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.560505 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1596e80225041099fb513ce477eae33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.560994 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 35428260e05e447ebf88ea41e63cdd02 (DEFAULT_TABLE table=default.Table_c5dbc5b105b8416c90d3792d29568b83 [id=a1839631f0c64b16a3f0411b514f93a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.561573 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35428260e05e447ebf88ea41e63cdd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.563107 32625 tablet_service.cc:1468] Processing CreateTablet for tablet 35428260e05e447ebf88ea41e63cdd02 (DEFAULT_TABLE table=default.Table_c5dbc5b105b8416c90d3792d29568b83 [id=a1839631f0c64b16a3f0411b514f93a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.563601 32625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35428260e05e447ebf88ea41e63cdd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.565237 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 5ca4957928fc4630aa8d8ca6110d7598 (DEFAULT_TABLE table=default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.566602 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f459b85abba44d288b0218c187f0098 with delete_type TABLET_DATA_DELETED (replaced by tablet c1596e80225041099fb513ce477eae33) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.567250 32631 tablet_service.cc:1468] Processing CreateTablet for tablet d67a8f1f570741ebb886c4d136ca40d2 (DEFAULT_TABLE table=default.Table_be622804eb114901b245383926c3afec [id=b020e49a601748e9bcbf397e235dc596]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.567679 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d67a8f1f570741ebb886c4d136ca40d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.569466 32355 tablet_service.cc:1468] Processing CreateTablet for tablet c1596e80225041099fb513ce477eae33 (DEFAULT_TABLE table=default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.570066 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca4957928fc4630aa8d8ca6110d7598. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.571105 32632 tablet_service.cc:1468] Processing CreateTablet for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b (DEFAULT_TABLE table=default.Table_0b402e459582438db7f3ad072b276ee0 [id=562e3d3edc3f4341ae8909d9fcf2e6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.571213 32352 tablet_service.cc:1468] Processing CreateTablet for tablet b6ad6cd629c4423ca997efb517cc975b (DEFAULT_TABLE table=default.Table_47264042125847ac997923572c293a3e [id=4214b1f0ca1744dc9b46d3f4d7618ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.571534 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.574661 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1596e80225041099fb513ce477eae33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.575836 32495 tablet_service.cc:1468] Processing CreateTablet for tablet c1596e80225041099fb513ce477eae33 (DEFAULT_TABLE table=default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.576097 32497 tablet_service.cc:1468] Processing CreateTablet for tablet b6ad6cd629c4423ca997efb517cc975b (DEFAULT_TABLE table=default.Table_47264042125847ac997923572c293a3e [id=4214b1f0ca1744dc9b46d3f4d7618ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.576221 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1596e80225041099fb513ce477eae33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.576447 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6ad6cd629c4423ca997efb517cc975b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.576763 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 35428260e05e447ebf88ea41e63cdd02 (DEFAULT_TABLE table=default.Table_c5dbc5b105b8416c90d3792d29568b83 [id=a1839631f0c64b16a3f0411b514f93a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.577093 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35428260e05e447ebf88ea41e63cdd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.578089 32353 tablet_service.cc:1468] Processing CreateTablet for tablet d67a8f1f570741ebb886c4d136ca40d2 (DEFAULT_TABLE table=default.Table_be622804eb114901b245383926c3afec [id=b020e49a601748e9bcbf397e235dc596]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.579568 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 5ca4957928fc4630aa8d8ca6110d7598 (DEFAULT_TABLE table=default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.579865 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca4957928fc4630aa8d8ca6110d7598. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.581351 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 5ca4957928fc4630aa8d8ca6110d7598 (DEFAULT_TABLE table=default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.581668 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ca4957928fc4630aa8d8ca6110d7598. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.582634 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b (DEFAULT_TABLE table=default.Table_0b402e459582438db7f3ad072b276ee0 [id=562e3d3edc3f4341ae8909d9fcf2e6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.585638 32624 tablet_service.cc:1468] Processing CreateTablet for tablet 0c77fd6e54904722acb2d1437c627ba8 (DEFAULT_TABLE table=default.Table_13b2a4a5337e4aeab1c2578c2cee28a3 [id=be9754ecb40a413d9940224adbd7e2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.586156 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c77fd6e54904722acb2d1437c627ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.586793 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6ad6cd629c4423ca997efb517cc975b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.587004 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 69e772df590747ff841be5f218e55acf (DEFAULT_TABLE table=default.Table_c3e871b94f3944d6b2b75626c3af94ec [id=a61a71f32ae444338b041f6e0a9fa286]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.587348 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e772df590747ff841be5f218e55acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.590865 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.594900 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d67a8f1f570741ebb886c4d136ca40d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.595624 341 ts_tablet_manager.cc:1905] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.595691 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 0c77fd6e54904722acb2d1437c627ba8 (DEFAULT_TABLE table=default.Table_13b2a4a5337e4aeab1c2578c2cee28a3 [id=be9754ecb40a413d9940224adbd7e2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.596051 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c77fd6e54904722acb2d1437c627ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.597673 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 04a3ae58d26142d58c4df4f52d7e3e2c (DEFAULT_TABLE table=default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.598050 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a3ae58d26142d58c4df4f52d7e3e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.598652 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 04a3ae58d26142d58c4df4f52d7e3e2c (DEFAULT_TABLE table=default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.599160 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a3ae58d26142d58c4df4f52d7e3e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.602809 32360 tablet_service.cc:1468] Processing CreateTablet for tablet 69e772df590747ff841be5f218e55acf (DEFAULT_TABLE table=default.Table_c3e871b94f3944d6b2b75626c3af94ec [id=a61a71f32ae444338b041f6e0a9fa286]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.603204 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e772df590747ff841be5f218e55acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.607081 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 0c77fd6e54904722acb2d1437c627ba8 (DEFAULT_TABLE table=default.Table_13b2a4a5337e4aeab1c2578c2cee28a3 [id=be9754ecb40a413d9940224adbd7e2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.607450 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c77fd6e54904722acb2d1437c627ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.612632 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_6b0157583afb475b863719603a5ba38d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.613307 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b0157583afb475b863719603a5ba38d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.614445 340 tablet_replica.cc:331] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:23.614634 340 raft_consensus.cc:2241] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.614843 340 raft_consensus.cc:2270] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.615694 340 ts_tablet_manager.cc:1905] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.616070 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_b353d2e453f94712941697298f2320b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.616691 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b353d2e453f94712941697298f2320b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.618237 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet b9d871507dcc46fc80f6e0ac739c6476 with delete_type TABLET_DATA_DELETED (replaced by tablet ca99c3b0cfec4aa481b620668b20947e) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.620107 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b6f3fe1b10143aa9b54022546812498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.627487 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_8a1e4cf5c8a54882b63f1dfd10533986"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.627720 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_c4287575a32b4847ba2379ef60074ba9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.627948 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_f82e868fa65a4e4fb9be937840796e7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.628134 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a1e4cf5c8a54882b63f1dfd10533986 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:23.628214 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4287575a32b4847ba2379ef60074ba9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:23.628314 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f82e868fa65a4e4fb9be937840796e7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.627523 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.628840 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.629587 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_89aa21f527bb4598bd99f4cc8eb5a5a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.630024 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89aa21f527bb4598bd99f4cc8eb5a5a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.631204 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_f6f76f6a3e724d6a8ba2e75dfe18557c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.631812 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6f76f6a3e724d6a8ba2e75dfe18557c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.631901 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 2eb99713f9214e6488d5c6e140ac263c (DEFAULT_TABLE table=default.Table_2807052f51c24614b4068e8113db2abd [id=9e90ae34db704c408759a60271e0bf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.632819 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eb99713f9214e6488d5c6e140ac263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.636288 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 04a3ae58d26142d58c4df4f52d7e3e2c (DEFAULT_TABLE table=default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.636780 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04a3ae58d26142d58c4df4f52d7e3e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.640098 32125 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_1ef4fa3350714732a2ecec291c255ed6" } modify_external_catalogs: true
I20250623 08:04:23.640287 32125 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_1ef4fa3350714732a2ecec291c255ed6" } modify_external_catalogs: true
I20250623 08:04:23.640882 340 ts_tablet_manager.cc:1918] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.641014 340 log.cc:1199] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b6ee90c0361c491b94ca9ff06962de1b
I20250623 08:04:23.641422 340 ts_tablet_manager.cc:1939] T b6ee90c0361c491b94ca9ff06962de1b P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:23.642519 340 tablet_replica.cc:331] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:23.642720 340 raft_consensus.cc:2241] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.642906 340 raft_consensus.cc:2270] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.643462 340 ts_tablet_manager.cc:1905] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.646382 341 ts_tablet_manager.cc:1918] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.647406 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 2eb99713f9214e6488d5c6e140ac263c (DEFAULT_TABLE table=default.Table_2807052f51c24614b4068e8113db2abd [id=9e90ae34db704c408759a60271e0bf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.647912 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eb99713f9214e6488d5c6e140ac263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.658576 340 ts_tablet_manager.cc:1918] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.658772 340 log.cc:1199] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/3c5e77c518e646e19e67c84a8f687801
I20250623 08:04:23.659147 340 ts_tablet_manager.cc:1939] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:23.659993 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 2eb99713f9214e6488d5c6e140ac263c (DEFAULT_TABLE table=default.Table_2807052f51c24614b4068e8113db2abd [id=9e90ae34db704c408759a60271e0bf82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.660403 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eb99713f9214e6488d5c6e140ac263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.660591 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet b6ee90c0361c491b94ca9ff06962de1b (table default.Table_985d82a3508246c7bec42eac00b838f0 [id=c967f23f2af84202a59d895e5fb7a2de]) successfully deleted
I20250623 08:04:23.662673 32128 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c3e871b94f3944d6b2b75626c3af94ec" } modify_external_catalogs: true
I20250623 08:04:23.662932 32128 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c3e871b94f3944d6b2b75626c3af94ec" } modify_external_catalogs: true
I20250623 08:04:23.665023 340 tablet_replica.cc:331] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:23.665273 340 raft_consensus.cc:2241] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.665524 340 raft_consensus.cc:2270] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.669121 340 ts_tablet_manager.cc:1905] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.671990 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b (DEFAULT_TABLE table=default.Table_0b402e459582438db7f3ad072b276ee0 [id=562e3d3edc3f4341ae8909d9fcf2e6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.672499 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.683864 32500 tablet_service.cc:1468] Processing CreateTablet for tablet d67a8f1f570741ebb886c4d136ca40d2 (DEFAULT_TABLE table=default.Table_be622804eb114901b245383926c3afec [id=b020e49a601748e9bcbf397e235dc596]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.684444 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d67a8f1f570741ebb886c4d136ca40d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.692782 339 tablet_bootstrap.cc:492] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.692750 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 3c5e77c518e646e19e67c84a8f687801 (table default.Table_25af89afee014e4bbecdb22a93780edc [id=9a792e7fecc84b6dab8eeced687eed5d]) successfully deleted
I20250623 08:04:23.693018 339 ts_tablet_manager.cc:1397] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.429s user 0.045s sys 0.071s
I20250623 08:04:23.693879 339 raft_consensus.cc:357] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.694113 339 raft_consensus.cc:383] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.694193 339 raft_consensus.cc:738] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.694432 339 consensus_queue.cc:260] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.699491 32122 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_cd5fd778975c47eb8aa915cb59bd588c" } new_table_name: "default.Table_2ae738ad8d0d430981d05bf2d7239d73" modify_external_catalogs: true
I20250623 08:04:23.700232 340 ts_tablet_manager.cc:1918] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.700368 340 log.cc:1199] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/c7d66b3414784129a5fd5941860666cd
I20250623 08:04:23.700702 340 ts_tablet_manager.cc:1939] T c7d66b3414784129a5fd5941860666cd P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:23.701041 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_40311f16d5ff467e9044de06e628afe9" } new_table_name: "default.Table_eefa8d393087436793f03c22b76ab152" modify_external_catalogs: true
I20250623 08:04:23.703792 324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 08:04:23.704946 324 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 08:04:23.705262 324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 08:04:23.706578 339 ts_tablet_manager.cc:1428] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250623 08:04:23.706976 339 tablet_bootstrap.cc:492] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.711006 339 tablet_bootstrap.cc:654] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.720459 32130 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_3f74ba855a0f49be9cbfbdcf3e715862" } modify_external_catalogs: true
I20250623 08:04:23.720685 32130 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_3f74ba855a0f49be9cbfbdcf3e715862" } modify_external_catalogs: true
I20250623 08:04:23.720786 32124 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_52580ab1940b40b0b31f4b95f9993f67" } modify_external_catalogs: true
I20250623 08:04:23.720953 32124 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_52580ab1940b40b0b31f4b95f9993f67" } modify_external_catalogs: true
I20250623 08:04:23.721180 32129 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_2807052f51c24614b4068e8113db2abd" } modify_external_catalogs: true
I20250623 08:04:23.721325 32129 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_2807052f51c24614b4068e8113db2abd" } modify_external_catalogs: true
I20250623 08:04:23.722034 32126 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_13b2a4a5337e4aeab1c2578c2cee28a3" } new_table_name: "default.Table_841dcc2d3ae642a29c4cf6c8fea37e88" modify_external_catalogs: true
I20250623 08:04:23.723084 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet c7d66b3414784129a5fd5941860666cd (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]) successfully deleted
W20250623 08:04:23.723465 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 0f459b85abba44d288b0218c187f0098 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f459b85abba44d288b0218c187f0098 already in progress: creating tablet
I20250623 08:04:23.723572 32125 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet abf018986c24473688c8e5eec724f8ee
W20250623 08:04:23.723827 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 9b6f3fe1b10143aa9b54022546812498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b6f3fe1b10143aa9b54022546812498 already in progress: creating tablet
W20250623 08:04:23.724190 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet c7385d86316f4a3ab68a77d18080ec20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7385d86316f4a3ab68a77d18080ec20 already in progress: creating tablet
W20250623 08:04:23.724464 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet ed5c81d9e56c4ee28546cedb9a579f31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed5c81d9e56c4ee28546cedb9a579f31 already in progress: creating tablet
I20250623 08:04:23.725944 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet abf018986c24473688c8e5eec724f8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.726678 32125 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_787076d19b96478889d932e380a68e59 [id=6cc9b03b74284f159ac54dd2dc10dc4b]). Expected version 2 got 0
I20250623 08:04:23.727521 341 log.cc:1199] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/c7d66b3414784129a5fd5941860666cd
I20250623 08:04:23.727900 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet abf018986c24473688c8e5eec724f8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.728042 341 ts_tablet_manager.cc:1939] T c7d66b3414784129a5fd5941860666cd P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:23.679271 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 69e772df590747ff841be5f218e55acf (DEFAULT_TABLE table=default.Table_c3e871b94f3944d6b2b75626c3af94ec [id=a61a71f32ae444338b041f6e0a9fa286]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.729548 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69e772df590747ff841be5f218e55acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.732198 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 69e772df590747ff841be5f218e55acf
I20250623 08:04:23.734189 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_a94b76793b7a49a687d0052141dd50c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.712869 343 tablet_bootstrap.cc:492] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:23.734692 343 ts_tablet_manager.cc:1397] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.341s user 0.020s sys 0.049s
W20250623 08:04:23.734877 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a94b76793b7a49a687d0052141dd50c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.735819 343 raft_consensus.cc:357] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.736122 343 raft_consensus.cc:383] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.736231 343 raft_consensus.cc:738] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.736495 343 consensus_queue.cc:260] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.725314 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet abf018986c24473688c8e5eec724f8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.738094 339 tablet_bootstrap.cc:492] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.738250 339 ts_tablet_manager.cc:1397] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.003s
I20250623 08:04:23.739082 339 raft_consensus.cc:357] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.739315 339 raft_consensus.cc:383] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.739406 339 raft_consensus.cc:738] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.739620 339 consensus_queue.cc:260] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.740078 339 ts_tablet_manager.cc:1428] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:23.740382 339 tablet_bootstrap.cc:492] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.745009 339 tablet_bootstrap.cc:654] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.747360 32121 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a564e263b16445d28147f335d686b132" } new_table_name: "default.Table_11b641dab5dd44b5b0ef1a309139d203" modify_external_catalogs: true
I20250623 08:04:23.749068 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_d72377a0c789492eaeaf570b4412d249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.749564 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d72377a0c789492eaeaf570b4412d249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.751997 342 tablet_bootstrap.cc:492] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.752197 342 ts_tablet_manager.cc:1397] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.406s user 0.075s sys 0.150s
I20250623 08:04:23.753214 342 raft_consensus.cc:357] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.753691 342 raft_consensus.cc:383] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.753800 342 raft_consensus.cc:738] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.757773 32123 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f82e868fa65a4e4fb9be937840796e7f" } new_table_name: "default.Table_5e8e88b2237f435595d619af47384b4e" modify_external_catalogs: true
I20250623 08:04:23.758337 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet 69e772df590747ff841be5f218e55acf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.759814 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c5e77c518e646e19e67c84a8f687801 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.760147 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 69e772df590747ff841be5f218e55acf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.760340 343 ts_tablet_manager.cc:1428] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250623 08:04:23.760408 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 69e772df590747ff841be5f218e55acf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.761096 343 tablet_bootstrap.cc:492] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:23.763180 342 consensus_queue.cc:260] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.764356 342 ts_tablet_manager.cc:1428] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250623 08:04:23.764670 342 tablet_bootstrap.cc:492] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:23.766301 32122 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1db513f73d24116af299da4d267aabf consensus state has no leader
I20250623 08:04:23.766841 342 tablet_bootstrap.cc:654] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:23.766997 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e90151394fb842498f4e780acab1f1aa consensus state has no leader
I20250623 08:04:23.767321 32127 master_service.cc:948] ReplaceTablet: received request to replace tablet abf018986c24473688c8e5eec724f8ee from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.770112 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet c7d66b3414784129a5fd5941860666cd (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]) successfully deleted
I20250623 08:04:23.770629 32122 master_service.cc:948] ReplaceTablet: received request to replace tablet a46c0a89fc014054a75ed198e824b5b6 from {username='slave'} at 127.0.0.1:50426
W20250623 08:04:23.777504 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet c7385d86316f4a3ab68a77d18080ec20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7385d86316f4a3ab68a77d18080ec20 already in progress: creating tablet
W20250623 08:04:23.777742 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet abf018986c24473688c8e5eec724f8ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abf018986c24473688c8e5eec724f8ee already in progress: creating tablet
W20250623 08:04:23.777949 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 69e772df590747ff841be5f218e55acf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69e772df590747ff841be5f218e55acf already in progress: creating tablet
W20250623 08:04:23.778146 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet abf018986c24473688c8e5eec724f8ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abf018986c24473688c8e5eec724f8ee already in progress: creating tablet
W20250623 08:04:23.778363 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 69e772df590747ff841be5f218e55acf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69e772df590747ff841be5f218e55acf already in progress: creating tablet
I20250623 08:04:23.779328 339 tablet_bootstrap.cc:492] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.779496 339 ts_tablet_manager.cc:1397] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.039s user 0.019s sys 0.009s
I20250623 08:04:23.780197 32127 master_service.cc:948] ReplaceTablet: received request to replace tablet ed5c81d9e56c4ee28546cedb9a579f31 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.780316 339 raft_consensus.cc:357] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.780529 339 raft_consensus.cc:383] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.780606 339 raft_consensus.cc:738] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.780843 339 consensus_queue.cc:260] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.781323 339 ts_tablet_manager.cc:1428] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:23.781633 339 tablet_bootstrap.cc:492] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.782234 341 tablet_replica.cc:331] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:23.788180 32127 master_service.cc:948] ReplaceTablet: received request to replace tablet 125784661a7b4e4ba6a710b950e004f8 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:23.791057 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c5e77c518e646e19e67c84a8f687801" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:23.791487 341 raft_consensus.cc:2241] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.791982 341 raft_consensus.cc:2270] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.792989 341 ts_tablet_manager.cc:1905] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:23.793478 32314 leader_election.cc:343] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:23.795607 339 tablet_bootstrap.cc:654] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.798666 343 tablet_bootstrap.cc:654] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.811551 339 tablet_bootstrap.cc:492] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.811723 339 ts_tablet_manager.cc:1397] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20250623 08:04:23.816455 339 raft_consensus.cc:357] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.816639 339 raft_consensus.cc:383] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.816741 339 raft_consensus.cc:738] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.817018 339 consensus_queue.cc:260] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.817552 339 ts_tablet_manager.cc:1428] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:23.817852 339 tablet_bootstrap.cc:492] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.819644 339 tablet_bootstrap.cc:654] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.823634 343 tablet_bootstrap.cc:492] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:23.823864 343 ts_tablet_manager.cc:1397] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.002s
I20250623 08:04:23.823976 342 tablet_bootstrap.cc:492] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.824133 342 ts_tablet_manager.cc:1397] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.009s
I20250623 08:04:23.824887 343 raft_consensus.cc:357] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.824980 342 raft_consensus.cc:357] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.825157 342 raft_consensus.cc:383] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.825189 343 raft_consensus.cc:383] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.825249 342 raft_consensus.cc:738] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.825455 343 raft_consensus.cc:738] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.825484 342 consensus_queue.cc:260] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.825996 342 ts_tablet_manager.cc:1428] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:23.826320 342 tablet_bootstrap.cc:492] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.828045 342 tablet_bootstrap.cc:654] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:23.828846 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet ed5c81d9e56c4ee28546cedb9a579f31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed5c81d9e56c4ee28546cedb9a579f31 already in progress: creating tablet
W20250623 08:04:23.829135 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 0f459b85abba44d288b0218c187f0098 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f459b85abba44d288b0218c187f0098 already in progress: creating tablet
W20250623 08:04:23.829559 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9b6f3fe1b10143aa9b54022546812498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b6f3fe1b10143aa9b54022546812498 already in progress: creating tablet
W20250623 08:04:23.829770 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet abf018986c24473688c8e5eec724f8ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abf018986c24473688c8e5eec724f8ee already in progress: creating tablet
I20250623 08:04:23.832612 343 consensus_queue.cc:260] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:23.833966 32121 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1e0c11ec768a43a48c6dc0f3dd974b71 consensus state has no leader
I20250623 08:04:23.834297 32121 master_service.cc:948] ReplaceTablet: received request to replace tablet 69e772df590747ff841be5f218e55acf from {username='slave'} at 127.0.0.1:50426
W20250623 08:04:23.834873 32126 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c77fd6e54904722acb2d1437c627ba8 consensus state has no leader
I20250623 08:04:23.835530 32124 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet e0c97148429046dc88213ff70a605c99
W20250623 08:04:23.834874 32123 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce1f3ef3690c4837b95b105733884f1f has no consensus state
I20250623 08:04:23.838598 343 ts_tablet_manager.cc:1428] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250623 08:04:23.839521 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet e0c97148429046dc88213ff70a605c99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.840332 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet e0c97148429046dc88213ff70a605c99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.841418 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet e0c97148429046dc88213ff70a605c99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.842196 343 tablet_bootstrap.cc:492] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:23.844328 343 tablet_bootstrap.cc:654] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.844813 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ee90c0361c491b94ca9ff06962de1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:22 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.848728 342 tablet_bootstrap.cc:492] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.848953 342 ts_tablet_manager.cc:1397] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.001s
I20250623 08:04:23.849840 342 raft_consensus.cc:357] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.850085 342 raft_consensus.cc:383] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.850207 342 raft_consensus.cc:738] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.850515 342 consensus_queue.cc:260] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.851220 342 ts_tablet_manager.cc:1428] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:23.853008 341 ts_tablet_manager.cc:1918] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.853164 341 log.cc:1199] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/3c5e77c518e646e19e67c84a8f687801
I20250623 08:04:23.853400 342 tablet_bootstrap.cc:492] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.853557 341 ts_tablet_manager.cc:1939] T 3c5e77c518e646e19e67c84a8f687801 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:23.854836 32314 leader_election.cc:343] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 3c5e77c518e646e19e67c84a8f687801
I20250623 08:04:23.855058 32314 leader_election.cc:304] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
W20250623 08:04:23.855489 32314 raft_consensus.cc:2684] T 3c5e77c518e646e19e67c84a8f687801 P 7351aa31800245659edc9837e27daffd: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:23.855758 342 tablet_bootstrap.cc:654] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.855779 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet a46c0a89fc014054a75ed198e824b5b6
I20250623 08:04:23.856267 32122 catalog_manager.cc:6652] ReplaceTablet: tablet a46c0a89fc014054a75ed198e824b5b6 deleted and replaced by tablet 228375592aee48a1aa21e1d16962b663
W20250623 08:04:23.859103 32116 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b6e29ab07df249748092a2f50a8c0273 consensus state has no leader
I20250623 08:04:23.861672 343 tablet_bootstrap.cc:492] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:23.861821 343 ts_tablet_manager.cc:1397] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
W20250623 08:04:23.862386 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet e0c97148429046dc88213ff70a605c99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0c97148429046dc88213ff70a605c99 already in progress: creating tablet
I20250623 08:04:23.862670 343 raft_consensus.cc:357] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.862968 343 raft_consensus.cc:383] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 08:04:23.862985 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet e0c97148429046dc88213ff70a605c99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0c97148429046dc88213ff70a605c99 already in progress: creating tablet
I20250623 08:04:23.863103 343 raft_consensus.cc:738] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.863539 343 consensus_queue.cc:260] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.864358 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet a46c0a89fc014054a75ed198e824b5b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 228375592aee48a1aa21e1d16962b663) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.864845 341 tablet_replica.cc:331] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:23.865026 341 raft_consensus.cc:2241] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.865202 341 raft_consensus.cc:2270] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.865799 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet a46c0a89fc014054a75ed198e824b5b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 228375592aee48a1aa21e1d16962b663) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.869385 343 ts_tablet_manager.cc:1428] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250623 08:04:23.870153 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet a46c0a89fc014054a75ed198e824b5b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 228375592aee48a1aa21e1d16962b663) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.871884 343 tablet_bootstrap.cc:492] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:23.872248 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 3c5e77c518e646e19e67c84a8f687801 (table default.Table_25af89afee014e4bbecdb22a93780edc [id=9a792e7fecc84b6dab8eeced687eed5d]) successfully deleted
W20250623 08:04:23.872764 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 69e772df590747ff841be5f218e55acf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69e772df590747ff841be5f218e55acf already in progress: creating tablet
W20250623 08:04:23.873405 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet e0c97148429046dc88213ff70a605c99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0c97148429046dc88213ff70a605c99 already in progress: creating tablet
I20250623 08:04:23.873759 343 tablet_bootstrap.cc:654] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:23.874035 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet a46c0a89fc014054a75ed198e824b5b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a46c0a89fc014054a75ed198e824b5b6 already in progress: creating tablet
I20250623 08:04:23.874944 342 tablet_bootstrap.cc:492] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:23.875087 342 ts_tablet_manager.cc:1397] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250623 08:04:23.875114 341 ts_tablet_manager.cc:1905] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.876057 342 raft_consensus.cc:357] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.876235 342 raft_consensus.cc:383] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.876349 342 raft_consensus.cc:738] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.876627 342 consensus_queue.cc:260] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.877144 342 ts_tablet_manager.cc:1428] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:23.877422 342 tablet_bootstrap.cc:492] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:23.879165 342 tablet_bootstrap.cc:654] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.881951 32128 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_787076d19b96478889d932e380a68e59 [id=6cc9b03b74284f159ac54dd2dc10dc4b]). Expected version 2 got 0
I20250623 08:04:23.882726 32124 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_787076d19b96478889d932e380a68e59 [id=6cc9b03b74284f159ac54dd2dc10dc4b]). Expected version 2 got 0
I20250623 08:04:23.886029 32130 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 75dde3beac4e47879c3f5e16151a4675
W20250623 08:04:23.892007 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet a46c0a89fc014054a75ed198e824b5b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a46c0a89fc014054a75ed198e824b5b6 already in progress: creating tablet
I20250623 08:04:23.893190 339 tablet_bootstrap.cc:492] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.893383 339 ts_tablet_manager.cc:1397] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.076s user 0.017s sys 0.013s
I20250623 08:04:23.894049 339 raft_consensus.cc:357] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.894198 339 raft_consensus.cc:383] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.894281 339 raft_consensus.cc:738] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.894486 339 consensus_queue.cc:260] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.894973 339 ts_tablet_manager.cc:1428] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250623 08:04:23.895078 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 (DEFAULT_TABLE table=default.Table_b353d2e453f94712941697298f2320b3 [id=b34c2a31f23742838fccd1bb73863d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.895313 339 tablet_bootstrap.cc:492] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.897006 339 tablet_bootstrap.cc:654] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.897039 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 (DEFAULT_TABLE table=default.Table_b353d2e453f94712941697298f2320b3 [id=b34c2a31f23742838fccd1bb73863d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.897533 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.897876 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 (DEFAULT_TABLE table=default.Table_b353d2e453f94712941697298f2320b3 [id=b34c2a31f23742838fccd1bb73863d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.898161 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.898212 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.902135 339 tablet_bootstrap.cc:492] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.902312 339 ts_tablet_manager.cc:1397] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20250623 08:04:23.903148 339 raft_consensus.cc:357] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.903342 339 raft_consensus.cc:383] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.903436 339 raft_consensus.cc:738] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.903637 339 consensus_queue.cc:260] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.904096 339 ts_tablet_manager.cc:1428] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:23.904387 339 tablet_bootstrap.cc:492] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.905977 339 tablet_bootstrap.cc:654] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.906741 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 0db9480bb07343fe935fe9bfc98bca77 (DEFAULT_TABLE table=default.Table_6b0157583afb475b863719603a5ba38d [id=d1fbcd8e89164d71b021f4b1f4223a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.907241 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db9480bb07343fe935fe9bfc98bca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.909775 339 tablet_bootstrap.cc:492] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.909948 339 ts_tablet_manager.cc:1397] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20250623 08:04:23.910698 339 raft_consensus.cc:357] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.910912 339 raft_consensus.cc:383] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.910974 339 raft_consensus.cc:738] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.911185 339 consensus_queue.cc:260] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.911669 339 ts_tablet_manager.cc:1428] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:23.911945 339 tablet_bootstrap.cc:492] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.912353 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_19e6011a07dd4ec08199244f47043017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.913066 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19e6011a07dd4ec08199244f47043017 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.913587 339 tablet_bootstrap.cc:654] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.916126 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_463abc7f8321496da24ccb06cba92d25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.916649 339 tablet_bootstrap.cc:492] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.916770 339 ts_tablet_manager.cc:1397] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
W20250623 08:04:23.916764 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_463abc7f8321496da24ccb06cba92d25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.917464 339 raft_consensus.cc:357] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.917616 339 raft_consensus.cc:383] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.917686 339 raft_consensus.cc:738] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.917876 339 consensus_queue.cc:260] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.918313 339 ts_tablet_manager.cc:1428] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:23.919919 339 tablet_bootstrap.cc:492] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.921655 339 tablet_bootstrap.cc:654] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.921896 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_ba6fcef0af3d4e1cb50fa5dca6559223"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.922531 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba6fcef0af3d4e1cb50fa5dca6559223 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.925222 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_d5617ea61e9d4010bf2ce97d61784515"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.925751 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5617ea61e9d4010bf2ce97d61784515 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.927006 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_26e019c7e20f450e8fb941854bbdc1e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.927627 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26e019c7e20f450e8fb941854bbdc1e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.929836 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_ba70febc0949477ba25d45eafa4f7b43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.930397 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba70febc0949477ba25d45eafa4f7b43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.937297 32125 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f6f76f6a3e724d6a8ba2e75dfe18557c" } modify_external_catalogs: true
I20250623 08:04:23.937506 32125 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f6f76f6a3e724d6a8ba2e75dfe18557c" } modify_external_catalogs: true
I20250623 08:04:23.938459 339 tablet_bootstrap.cc:492] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:23.938601 339 ts_tablet_manager.cc:1397] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20250623 08:04:23.938854 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:23.939328 339 raft_consensus.cc:357] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:23.939450 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.939563 339 raft_consensus.cc:383] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.939626 339 raft_consensus.cc:738] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.939838 339 consensus_queue.cc:260] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.941239 339 ts_tablet_manager.cc:1428] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:23.941584 339 tablet_bootstrap.cc:492] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:23.944164 339 tablet_bootstrap.cc:654] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.948866 343 tablet_bootstrap.cc:492] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:23.949077 343 ts_tablet_manager.cc:1397] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.003s
I20250623 08:04:23.949681 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 0db9480bb07343fe935fe9bfc98bca77 (DEFAULT_TABLE table=default.Table_6b0157583afb475b863719603a5ba38d [id=d1fbcd8e89164d71b021f4b1f4223a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.951753 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db9480bb07343fe935fe9bfc98bca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.951839 341 ts_tablet_manager.cc:1918] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.952028 341 log.cc:1199] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b6ee90c0361c491b94ca9ff06962de1b
I20250623 08:04:23.952505 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 75dde3beac4e47879c3f5e16151a4675 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:23.952589 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 0db9480bb07343fe935fe9bfc98bca77 (DEFAULT_TABLE table=default.Table_6b0157583afb475b863719603a5ba38d [id=d1fbcd8e89164d71b021f4b1f4223a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:23.952791 340 tablet_replica.cc:331] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:23.952944 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db9480bb07343fe935fe9bfc98bca77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:23.953025 340 raft_consensus.cc:2241] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.953202 340 raft_consensus.cc:2270] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.954031 340 ts_tablet_manager.cc:1905] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.956297 343 raft_consensus.cc:357] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.956475 343 raft_consensus.cc:383] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:23.956564 343 raft_consensus.cc:738] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:23.956761 343 consensus_queue.cc:260] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:23.957212 343 ts_tablet_manager.cc:1428] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250623 08:04:23.962002 340 ts_tablet_manager.cc:1918] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.962312 340 log.cc:1199] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/75dde3beac4e47879c3f5e16151a4675
I20250623 08:04:23.962839 341 ts_tablet_manager.cc:1939] T b6ee90c0361c491b94ca9ff06962de1b P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:23.962865 340 ts_tablet_manager.cc:1939] T 75dde3beac4e47879c3f5e16151a4675 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:23.967227 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 75dde3beac4e47879c3f5e16151a4675 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:23.967420 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 75dde3beac4e47879c3f5e16151a4675 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:23.967582 344 tablet_replica.cc:331] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:23.967789 344 raft_consensus.cc:2241] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.967986 344 raft_consensus.cc:2270] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.968022 341 tablet_replica.cc:331] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:23.968427 341 raft_consensus.cc:2241] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:23.968612 341 raft_consensus.cc:2270] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:23.968629 344 ts_tablet_manager.cc:1905] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.968919 343 tablet_bootstrap.cc:492] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:23.972398 341 ts_tablet_manager.cc:1905] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:23.974162 343 tablet_bootstrap.cc:654] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:23.977150 344 ts_tablet_manager.cc:1918] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.977351 344 log.cc:1199] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/75dde3beac4e47879c3f5e16151a4675
I20250623 08:04:23.977900 344 ts_tablet_manager.cc:1939] T 75dde3beac4e47879c3f5e16151a4675 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:23.981851 341 ts_tablet_manager.cc:1918] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:23.982016 341 log.cc:1199] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/75dde3beac4e47879c3f5e16151a4675
I20250623 08:04:23.982434 341 ts_tablet_manager.cc:1939] T 75dde3beac4e47879c3f5e16151a4675 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:23.986948 32127 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 125784661a7b4e4ba6a710b950e004f8
I20250623 08:04:23.987517 32127 catalog_manager.cc:6652] ReplaceTablet: tablet 125784661a7b4e4ba6a710b950e004f8 deleted and replaced by tablet ae7c2ac546114a819a543f1667fb7844
I20250623 08:04:23.989297 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_6251401a747d429cac1a9566cdaea5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:23.989959 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6251401a747d429cac1a9566cdaea5e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:23.991030 32129 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 2eb99713f9214e6488d5c6e140ac263c
I20250623 08:04:23.991833 32129 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c5dbc5b105b8416c90d3792d29568b83" } modify_external_catalogs: true
I20250623 08:04:23.992048 32129 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c5dbc5b105b8416c90d3792d29568b83" } modify_external_catalogs: true
I20250623 08:04:24.000130 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b6ee90c0361c491b94ca9ff06962de1b (table default.Table_985d82a3508246c7bec42eac00b838f0 [id=c967f23f2af84202a59d895e5fb7a2de]) successfully deleted
W20250623 08:04:24.000481 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet a46c0a89fc014054a75ed198e824b5b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a46c0a89fc014054a75ed198e824b5b6 already in progress: creating tablet
I20250623 08:04:24.001165 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 75dde3beac4e47879c3f5e16151a4675 (table default.Table_3f74ba855a0f49be9cbfbdcf3e715862 [id=26c5fff9eaea4788844a55e87ada7215]) successfully deleted
I20250623 08:04:24.001813 362 raft_consensus.cc:491] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.001986 362 raft_consensus.cc:513] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.002651 362 leader_election.cc:290] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.003067 343 tablet_bootstrap.cc:492] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.003216 343 ts_tablet_manager.cc:1397] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.003s
I20250623 08:04:24.003580 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f459b85abba44d288b0218c187f0098" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:24.003911 343 raft_consensus.cc:357] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.004096 343 raft_consensus.cc:383] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.004153 343 raft_consensus.cc:738] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.004071 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f459b85abba44d288b0218c187f0098" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:24.004387 343 consensus_queue.cc:260] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:24.005187 32314 leader_election.cc:343] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:24.005609 32314 leader_election.cc:343] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.005749 32314 leader_election.cc:304] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:24.006434 362 raft_consensus.cc:2747] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:24.007437 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 75dde3beac4e47879c3f5e16151a4675 (table default.Table_3f74ba855a0f49be9cbfbdcf3e715862 [id=26c5fff9eaea4788844a55e87ada7215]) successfully deleted
I20250623 08:04:24.007820 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 75dde3beac4e47879c3f5e16151a4675 (table default.Table_3f74ba855a0f49be9cbfbdcf3e715862 [id=26c5fff9eaea4788844a55e87ada7215]) successfully deleted
I20250623 08:04:24.008049 343 ts_tablet_manager.cc:1428] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:24.008600 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet 125784661a7b4e4ba6a710b950e004f8 with delete_type TABLET_DATA_DELETED (replaced by tablet ae7c2ac546114a819a543f1667fb7844) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.009840 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 125784661a7b4e4ba6a710b950e004f8 with delete_type TABLET_DATA_DELETED (replaced by tablet ae7c2ac546114a819a543f1667fb7844) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.010174 342 tablet_bootstrap.cc:492] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:24.010397 342 ts_tablet_manager.cc:1397] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.133s user 0.021s sys 0.033s
I20250623 08:04:24.010488 343 tablet_bootstrap.cc:492] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.010869 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 125784661a7b4e4ba6a710b950e004f8 with delete_type TABLET_DATA_DELETED (replaced by tablet ae7c2ac546114a819a543f1667fb7844) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.011458 342 raft_consensus.cc:357] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.011744 342 raft_consensus.cc:383] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.011801 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bf5243f44c043a0a1e9bc441fc91e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.011914 342 raft_consensus.cc:738] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.012234 342 consensus_queue.cc:260] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.012385 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet 2eb99713f9214e6488d5c6e140ac263c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.012635 343 tablet_bootstrap.cc:654] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.012899 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 2eb99713f9214e6488d5c6e140ac263c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.013774 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 2eb99713f9214e6488d5c6e140ac263c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.014562 341 tablet_replica.cc:331] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:24.014853 341 raft_consensus.cc:2241] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.014577 342 ts_tablet_manager.cc:1428] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250623 08:04:24.015143 341 raft_consensus.cc:2270] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.016337 342 tablet_bootstrap.cc:492] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:24.019784 342 tablet_bootstrap.cc:654] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.023164 341 ts_tablet_manager.cc:1905] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.030833 342 tablet_bootstrap.cc:492] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:24.031543 342 ts_tablet_manager.cc:1397] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20250623 08:04:24.031682 343 tablet_bootstrap.cc:492] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.031862 343 ts_tablet_manager.cc:1397] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250623 08:04:24.032281 362 raft_consensus.cc:491] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.032464 362 raft_consensus.cc:513] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.032829 343 raft_consensus.cc:357] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.033049 343 raft_consensus.cc:383] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.033146 343 raft_consensus.cc:738] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.033149 362 leader_election.cc:290] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.033459 343 consensus_queue.cc:260] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.034258 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6e29ab07df249748092a2f50a8c0273" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:24.034938 32314 leader_election.cc:343] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.035390 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6e29ab07df249748092a2f50a8c0273" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:24.035701 32520 raft_consensus.cc:2466] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:24.036340 342 raft_consensus.cc:357] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.036500 342 raft_consensus.cc:383] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.036612 342 raft_consensus.cc:738] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.036895 343 ts_tablet_manager.cc:1428] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250623 08:04:24.036835 342 consensus_queue.cc:260] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.037542 32314 leader_election.cc:304] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:24.037683 32128 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]). Expected version 1 got 0
I20250623 08:04:24.037835 362 raft_consensus.cc:2802] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:24.037933 362 raft_consensus.cc:491] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:24.038048 362 raft_consensus.cc:3058] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:24.040233 343 tablet_bootstrap.cc:492] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.041231 362 raft_consensus.cc:513] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.042027 32121 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_787076d19b96478889d932e380a68e59" } new_table_name: "default.Table_3cf27fe1edb94ead83092ffda8c5f7f1" modify_external_catalogs: true
I20250623 08:04:24.042825 343 tablet_bootstrap.cc:654] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.042892 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6e29ab07df249748092a2f50a8c0273" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:24.043181 32123 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_89aa21f527bb4598bd99f4cc8eb5a5a6" } new_table_name: "default.Table_86a910b4ef524f2a84b98f25141cd248" modify_external_catalogs: true
W20250623 08:04:24.043426 32314 leader_election.cc:343] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.044155 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6e29ab07df249748092a2f50a8c0273" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:24.044353 32520 raft_consensus.cc:3058] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:24.044662 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet 9b6f3fe1b10143aa9b54022546812498 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.045084 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet b1db513f73d24116af299da4d267aabf from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.046211 32125 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f5b5638d72924baf963bec7f3c873222
I20250623 08:04:24.047426 32520 raft_consensus.cc:2466] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:24.047642 341 ts_tablet_manager.cc:1918] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.047775 341 log.cc:1199] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/3bf5243f44c043a0a1e9bc441fc91e70
I20250623 08:04:24.047832 32314 leader_election.cc:304] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:24.048296 341 ts_tablet_manager.cc:1939] T 3bf5243f44c043a0a1e9bc441fc91e70 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:24.048660 362 leader_election.cc:290] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.048831 362 raft_consensus.cc:2802] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:24.048985 362 raft_consensus.cc:695] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:24.049408 362 consensus_queue.cc:237] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.050122 32129 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 35428260e05e447ebf88ea41e63cdd02
I20250623 08:04:24.052891 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_c09ee28ff44c4c43835c2faa880a1c86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.053452 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c09ee28ff44c4c43835c2faa880a1c86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.055011 32125 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_2ab537bbc9724fd69812910c17dca994" } new_table_name: "default.Table_25ef245b7f9344c0ad2e841fa78d0be7" modify_external_catalogs: true
I20250623 08:04:24.056222 343 tablet_bootstrap.cc:492] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.056404 343 ts_tablet_manager.cc:1397] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250623 08:04:24.057063 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d72377a0c789492eaeaf570b4412d249" } new_table_name: "default.Table_f9ba2bae4aad4caa9c87e89dc65d3dc5" modify_external_catalogs: true
I20250623 08:04:24.057329 343 raft_consensus.cc:357] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.057530 343 raft_consensus.cc:383] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.057618 343 raft_consensus.cc:738] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.057947 343 consensus_queue.cc:260] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.060020 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_8cc5ec1f97f94a85bb54e58eb38760ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.060652 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cc5ec1f97f94a85bb54e58eb38760ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.062609 343 ts_tablet_manager.cc:1428] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20250623 08:04:24.066443 343 tablet_bootstrap.cc:492] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.068835 343 tablet_bootstrap.cc:654] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.069846 32130 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_2ae738ad8d0d430981d05bf2d7239d73" } modify_external_catalogs: true
I20250623 08:04:24.070040 32130 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_2ae738ad8d0d430981d05bf2d7239d73" } modify_external_catalogs: true
I20250623 08:04:24.072607 32128 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_b353d2e453f94712941697298f2320b3" } new_table_name: "default.Table_f9b870e59836470094157d5e22d85338" modify_external_catalogs: true
I20250623 08:04:24.079495 342 ts_tablet_manager.cc:1428] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20250623 08:04:24.079972 342 tablet_bootstrap.cc:492] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:24.083200 342 tablet_bootstrap.cc:654] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:24.084596 32114 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 55893.1us
W20250623 08:04:24.085131 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 125784661a7b4e4ba6a710b950e004f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 125784661a7b4e4ba6a710b950e004f8 already in progress: creating tablet
I20250623 08:04:24.085532 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 3bf5243f44c043a0a1e9bc441fc91e70 (table default.Table_13b0e15543044cd58b65937ca08d8df4 [id=adb0245ebf874699887e167d5b46d1d2]) successfully deleted
W20250623 08:04:24.085876 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 2eb99713f9214e6488d5c6e140ac263c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2eb99713f9214e6488d5c6e140ac263c already in progress: creating tablet
W20250623 08:04:24.086170 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 125784661a7b4e4ba6a710b950e004f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 125784661a7b4e4ba6a710b950e004f8 already in progress: creating tablet
W20250623 08:04:24.086429 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 2eb99713f9214e6488d5c6e140ac263c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2eb99713f9214e6488d5c6e140ac263c already in progress: creating tablet
W20250623 08:04:24.086673 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 125784661a7b4e4ba6a710b950e004f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 125784661a7b4e4ba6a710b950e004f8 already in progress: creating tablet
W20250623 08:04:24.086977 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 2eb99713f9214e6488d5c6e140ac263c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2eb99713f9214e6488d5c6e140ac263c already in progress: creating tablet
I20250623 08:04:24.087666 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c7d30889c024c84b6585f007504139e with delete_type TABLET_DATA_DELETED (replaced by tablet b9d871507dcc46fc80f6e0ac739c6476) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.087999 341 tablet_replica.cc:331] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:24.088284 341 raft_consensus.cc:2241] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.088424 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet 35428260e05e447ebf88ea41e63cdd02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.088524 341 raft_consensus.cc:2270] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.089030 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 35428260e05e447ebf88ea41e63cdd02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.089732 341 ts_tablet_manager.cc:1905] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.090458 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 35428260e05e447ebf88ea41e63cdd02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.091398 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b5a56fe461641bea2bb269bf0564415 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.092041 340 tablet_replica.cc:331] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:24.092216 342 tablet_bootstrap.cc:492] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:24.092341 340 raft_consensus.cc:2241] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.092379 342 ts_tablet_manager.cc:1397] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20250623 08:04:24.092604 340 raft_consensus.cc:2270] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.093328 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a3c5a446184329b55fcc2cbf969831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.093420 340 ts_tablet_manager.cc:1905] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.093911 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f459b85abba44d288b0218c187f0098 with delete_type TABLET_DATA_DELETED (replaced by tablet c1596e80225041099fb513ce477eae33) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.094511 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c7d30889c024c84b6585f007504139e with delete_type TABLET_DATA_DELETED (replaced by tablet b9d871507dcc46fc80f6e0ac739c6476) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.095139 342 raft_consensus.cc:357] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.095217 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet a3bd7fa007544af69b3628911dca3a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.095358 342 raft_consensus.cc:383] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.095440 342 raft_consensus.cc:738] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.095984 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a3c5a446184329b55fcc2cbf969831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.096300 344 tablet_replica.cc:331] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:24.096556 344 raft_consensus.cc:2241] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.096782 344 raft_consensus.cc:2270] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.096885 341 ts_tablet_manager.cc:1918] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.097029 341 log.cc:1199] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/5c7d30889c024c84b6585f007504139e
I20250623 08:04:24.097395 341 ts_tablet_manager.cc:1939] T 5c7d30889c024c84b6585f007504139e P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:24.097651 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet a3bd7fa007544af69b3628911dca3a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.097746 344 ts_tablet_manager.cc:1905] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.098265 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b5a56fe461641bea2bb269bf0564415 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.099371 343 tablet_bootstrap.cc:492] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.099498 343 ts_tablet_manager.cc:1397] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.010s
I20250623 08:04:24.099889 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bf5243f44c043a0a1e9bc441fc91e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.099977 340 ts_tablet_manager.cc:1918] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.100100 340 log.cc:1199] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/9b5a56fe461641bea2bb269bf0564415
I20250623 08:04:24.100340 343 raft_consensus.cc:357] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.100485 340 ts_tablet_manager.cc:1939] T 9b5a56fe461641bea2bb269bf0564415 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:24.100539 343 raft_consensus.cc:383] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.100628 343 raft_consensus.cc:738] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.100843 343 consensus_queue.cc:260] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.102929 340 tablet_replica.cc:331] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:24.103125 340 raft_consensus.cc:2241] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.103261 340 raft_consensus.cc:2270] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.103967 343 ts_tablet_manager.cc:1428] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:24.104542 340 ts_tablet_manager.cc:1905] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.105054 343 tablet_bootstrap.cc:492] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.105602 344 ts_tablet_manager.cc:1918] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.105726 344 log.cc:1199] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/d3a3c5a446184329b55fcc2cbf969831
I20250623 08:04:24.106065 344 ts_tablet_manager.cc:1939] T d3a3c5a446184329b55fcc2cbf969831 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:24.106611 343 tablet_bootstrap.cc:654] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.107028 344 tablet_replica.cc:331] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:24.107164 344 raft_consensus.cc:2241] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.107292 344 raft_consensus.cc:2270] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.108320 344 ts_tablet_manager.cc:1905] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.114318 340 ts_tablet_manager.cc:1918] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.114473 340 log.cc:1199] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/d3a3c5a446184329b55fcc2cbf969831
I20250623 08:04:24.114831 340 ts_tablet_manager.cc:1939] T d3a3c5a446184329b55fcc2cbf969831 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:24.117889 344 ts_tablet_manager.cc:1918] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.118011 344 log.cc:1199] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a3bd7fa007544af69b3628911dca3a80
I20250623 08:04:24.118389 344 ts_tablet_manager.cc:1939] T a3bd7fa007544af69b3628911dca3a80 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:24.118388 343 tablet_bootstrap.cc:492] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.118583 343 ts_tablet_manager.cc:1397] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20250623 08:04:24.119464 340 tablet_replica.cc:331] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:24.119494 343 raft_consensus.cc:357] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.119659 340 raft_consensus.cc:2241] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.119690 343 raft_consensus.cc:383] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.119755 343 raft_consensus.cc:738] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.119828 340 raft_consensus.cc:2270] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.120021 343 consensus_queue.cc:260] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.120498 340 ts_tablet_manager.cc:1905] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.123095 343 ts_tablet_manager.cc:1428] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250623 08:04:24.123907 344 tablet_replica.cc:331] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:24.124094 344 raft_consensus.cc:2241] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.124825 344 raft_consensus.cc:2270] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:24.125478 32123 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ac9f38877b24d619c280a1aaa0b7b08 has no consensus state
I20250623 08:04:24.125835 343 tablet_bootstrap.cc:492] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.125813 32123 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c77fd6e54904722acb2d1437c627ba8 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.126295 32124 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet b1db513f73d24116af299da4d267aabf
I20250623 08:04:24.126583 340 ts_tablet_manager.cc:1918] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.126775 340 log.cc:1199] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/0f459b85abba44d288b0218c187f0098
I20250623 08:04:24.126793 32124 catalog_manager.cc:6652] ReplaceTablet: tablet b1db513f73d24116af299da4d267aabf deleted and replaced by tablet 776b412b3f564a7581c6a5e1c84e8499
I20250623 08:04:24.127182 340 ts_tablet_manager.cc:1939] T 0f459b85abba44d288b0218c187f0098 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:24.127290 32124 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_6251401a747d429cac1a9566cdaea5e8" } modify_external_catalogs: true
I20250623 08:04:24.127491 32124 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_6251401a747d429cac1a9566cdaea5e8" } modify_external_catalogs: true
I20250623 08:04:24.127642 343 tablet_bootstrap.cc:654] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.128160 340 tablet_replica.cc:331] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:24.128347 340 raft_consensus.cc:2241] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.128520 340 raft_consensus.cc:2270] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.129237 344 ts_tablet_manager.cc:1905] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.129246 32126 master_service.cc:948] ReplaceTablet: received request to replace tablet b9d871507dcc46fc80f6e0ac739c6476 from {username='slave'} at 127.0.0.1:50426
W20250623 08:04:24.130089 32125 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7300f526d91f441bb164fe47f3a4f0ce consensus state has no leader
W20250623 08:04:24.130537 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c01121f2e644d6aa6e8ec228bf75be9 has no consensus state
I20250623 08:04:24.135489 340 ts_tablet_manager.cc:1905] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.136271 343 tablet_bootstrap.cc:492] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.136410 343 ts_tablet_manager.cc:1397] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250623 08:04:24.137192 343 raft_consensus.cc:357] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.137367 343 raft_consensus.cc:383] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.137509 343 raft_consensus.cc:738] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:24.137835 32128 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 consensus state has no leader
I20250623 08:04:24.137792 343 consensus_queue.cc:260] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.141368 343 ts_tablet_manager.cc:1428] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:24.143682 344 ts_tablet_manager.cc:1918] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.143836 344 log.cc:1199] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9b5a56fe461641bea2bb269bf0564415
I20250623 08:04:24.144214 344 ts_tablet_manager.cc:1939] T 9b5a56fe461641bea2bb269bf0564415 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:24.144372 343 tablet_bootstrap.cc:492] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.145262 340 ts_tablet_manager.cc:1918] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.145411 340 log.cc:1199] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/5c7d30889c024c84b6585f007504139e
I20250623 08:04:24.145721 340 ts_tablet_manager.cc:1939] T 5c7d30889c024c84b6585f007504139e P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:24.146020 343 tablet_bootstrap.cc:654] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.146878 340 tablet_replica.cc:331] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:24.147121 340 raft_consensus.cc:2241] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.147334 340 raft_consensus.cc:2270] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.096478 342 consensus_queue.cc:260] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.147962 342 ts_tablet_manager.cc:1428] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.055s user 0.001s sys 0.001s
I20250623 08:04:24.148103 340 ts_tablet_manager.cc:1905] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.148283 342 tablet_bootstrap.cc:492] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:24.148392 339 tablet_bootstrap.cc:492] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:24.148538 339 ts_tablet_manager.cc:1397] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.207s user 0.044s sys 0.058s
I20250623 08:04:24.149379 339 raft_consensus.cc:357] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.149569 339 raft_consensus.cc:383] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.149670 339 raft_consensus.cc:738] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.149930 339 consensus_queue.cc:260] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.150260 342 tablet_bootstrap.cc:654] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.150458 339 ts_tablet_manager.cc:1428] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:24.150771 339 tablet_bootstrap.cc:492] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:24.152503 339 tablet_bootstrap.cc:654] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.156174 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 5c7d30889c024c84b6585f007504139e (table default.Table_d27db7e0fea34b5182353dee8ae23cac [id=816747bd5e004ae5964b7b885aa249fb]) successfully deleted
W20250623 08:04:24.157410 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 35428260e05e447ebf88ea41e63cdd02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35428260e05e447ebf88ea41e63cdd02 already in progress: creating tablet
W20250623 08:04:24.158071 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 35428260e05e447ebf88ea41e63cdd02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35428260e05e447ebf88ea41e63cdd02 already in progress: creating tablet
I20250623 08:04:24.158711 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 9b5a56fe461641bea2bb269bf0564415 (table default.Table_fac8d606d7454d3184af3a8d4c20ccc4 [id=d94a8f840a0c48a6bdfdca2ff5f0c5dc]) successfully deleted
I20250623 08:04:24.158828 340 ts_tablet_manager.cc:1918] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.158963 340 log.cc:1199] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/a3bd7fa007544af69b3628911dca3a80
I20250623 08:04:24.159209 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet d3a3c5a446184329b55fcc2cbf969831 (table default.Table_aeb7585a9aad4577b59b1b1164aa016e [id=cf9d65382d8b4fd98f18c628c2ccad23]) successfully deleted
I20250623 08:04:24.159512 340 ts_tablet_manager.cc:1939] T a3bd7fa007544af69b3628911dca3a80 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:24.159813 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 0f459b85abba44d288b0218c187f0098 (table default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]) successfully deleted
I20250623 08:04:24.160136 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 5c7d30889c024c84b6585f007504139e (table default.Table_d27db7e0fea34b5182353dee8ae23cac [id=816747bd5e004ae5964b7b885aa249fb]) successfully deleted
W20250623 08:04:24.160439 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 35428260e05e447ebf88ea41e63cdd02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35428260e05e447ebf88ea41e63cdd02 already in progress: creating tablet
I20250623 08:04:24.160660 340 tablet_replica.cc:331] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:24.160737 339 tablet_bootstrap.cc:492] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:24.160878 340 raft_consensus.cc:2241] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.160900 339 ts_tablet_manager.cc:1397] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20250623 08:04:24.161027 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet d3a3c5a446184329b55fcc2cbf969831 (table default.Table_aeb7585a9aad4577b59b1b1164aa016e [id=cf9d65382d8b4fd98f18c628c2ccad23]) successfully deleted
I20250623 08:04:24.161329 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet a3bd7fa007544af69b3628911dca3a80 (table default.Table_0beadbd550664534bbf90d51ca7a4b0b [id=967a9b35621a401681cb78fb66402525]) successfully deleted
I20250623 08:04:24.161612 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 9b5a56fe461641bea2bb269bf0564415 (table default.Table_fac8d606d7454d3184af3a8d4c20ccc4 [id=d94a8f840a0c48a6bdfdca2ff5f0c5dc]) successfully deleted
I20250623 08:04:24.161804 339 raft_consensus.cc:357] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.162045 339 raft_consensus.cc:383] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.162128 339 raft_consensus.cc:738] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.162428 339 consensus_queue.cc:260] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.162792 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet b1db513f73d24116af299da4d267aabf with delete_type TABLET_DATA_DELETED (replaced by tablet 776b412b3f564a7581c6a5e1c84e8499) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.163046 339 ts_tablet_manager.cc:1428] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:24.163378 339 tablet_bootstrap.cc:492] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:24.164592 342 tablet_bootstrap.cc:492] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:24.164762 342 ts_tablet_manager.cc:1397] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20250623 08:04:24.164963 340 raft_consensus.cc:2270] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.165522 342 raft_consensus.cc:357] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.165737 339 tablet_bootstrap.cc:654] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.165729 342 raft_consensus.cc:383] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.165824 342 raft_consensus.cc:738] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.165743 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet b1db513f73d24116af299da4d267aabf with delete_type TABLET_DATA_DELETED (replaced by tablet 776b412b3f564a7581c6a5e1c84e8499) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.166115 342 consensus_queue.cc:260] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.166970 340 ts_tablet_manager.cc:1905] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.167129 342 ts_tablet_manager.cc:1428] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:24.167448 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet b1db513f73d24116af299da4d267aabf with delete_type TABLET_DATA_DELETED (replaced by tablet 776b412b3f564a7581c6a5e1c84e8499) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.167589 343 tablet_bootstrap.cc:492] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.167762 343 ts_tablet_manager.cc:1397] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.005s
I20250623 08:04:24.168705 343 raft_consensus.cc:357] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.169343 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet b9d871507dcc46fc80f6e0ac739c6476 with delete_type TABLET_DATA_DELETED (replaced by tablet ca99c3b0cfec4aa481b620668b20947e) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.169536 343 raft_consensus.cc:383] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.169642 343 raft_consensus.cc:738] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.169672 344 tablet_replica.cc:331] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:24.169868 344 raft_consensus.cc:2241] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.169899 343 consensus_queue.cc:260] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.170082 344 raft_consensus.cc:2270] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.170419 343 ts_tablet_manager.cc:1428] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:24.170668 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet a3bd7fa007544af69b3628911dca3a80 (table default.Table_0beadbd550664534bbf90d51ca7a4b0b [id=967a9b35621a401681cb78fb66402525]) successfully deleted
I20250623 08:04:24.170732 343 tablet_bootstrap.cc:492] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.173148 343 tablet_bootstrap.cc:654] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.175745 344 ts_tablet_manager.cc:1905] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.178072 339 tablet_bootstrap.cc:492] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:24.178207 339 ts_tablet_manager.cc:1397] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20250623 08:04:24.179339 339 raft_consensus.cc:357] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.179520 339 raft_consensus.cc:383] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.179622 339 raft_consensus.cc:738] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.179868 339 consensus_queue.cc:260] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:24.180063 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet b1db513f73d24116af299da4d267aabf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1db513f73d24116af299da4d267aabf already in progress: creating tablet
I20250623 08:04:24.181526 339 ts_tablet_manager.cc:1428] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:24.181807 339 tablet_bootstrap.cc:492] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:24.182039 344 ts_tablet_manager.cc:1918] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.182137 344 log.cc:1199] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b9d871507dcc46fc80f6e0ac739c6476
I20250623 08:04:24.182577 344 ts_tablet_manager.cc:1939] T b9d871507dcc46fc80f6e0ac739c6476 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:24.183295 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f459b85abba44d288b0218c187f0098 with delete_type TABLET_DATA_DELETED (replaced by tablet c1596e80225041099fb513ce477eae33) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.184355 342 tablet_bootstrap.cc:492] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:24.184790 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b1db513f73d24116af299da4d267aabf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1db513f73d24116af299da4d267aabf already in progress: creating tablet
I20250623 08:04:24.185096 339 tablet_bootstrap.cc:654] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.186115 342 tablet_bootstrap.cc:654] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.187088 344 tablet_replica.cc:331] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:24.187222 343 tablet_bootstrap.cc:492] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.187296 344 raft_consensus.cc:2241] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.187369 343 ts_tablet_manager.cc:1397] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20250623 08:04:24.187462 344 raft_consensus.cc:2270] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.188218 344 ts_tablet_manager.cc:1905] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.188164 343 raft_consensus.cc:357] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.188426 343 raft_consensus.cc:383] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.188516 343 raft_consensus.cc:738] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.189141 32124 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet b2841599c70646b3be0f48f6ca72c4dc
I20250623 08:04:24.190604 343 consensus_queue.cc:260] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.193964 343 ts_tablet_manager.cc:1428] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:24.195775 343 tablet_bootstrap.cc:492] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.196601 339 tablet_bootstrap.cc:492] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:24.196556 32122 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 1e0c11ec768a43a48c6dc0f3dd974b71 (table default.Table_11b641dab5dd44b5b0ef1a309139d203 [id=bb849e86d7084959830c68207fb99061]). Expected version 2 got 0
I20250623 08:04:24.196763 339 ts_tablet_manager.cc:1397] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20250623 08:04:24.197494 339 raft_consensus.cc:357] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.197706 339 raft_consensus.cc:383] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.197780 339 raft_consensus.cc:738] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.197809 343 tablet_bootstrap.cc:654] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.197883 32129 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_787076d19b96478889d932e380a68e59 [id=6cc9b03b74284f159ac54dd2dc10dc4b]). Expected version 2 got 0
I20250623 08:04:24.198045 339 consensus_queue.cc:260] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.198714 339 ts_tablet_manager.cc:1428] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:24.199123 339 tablet_bootstrap.cc:492] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:24.200011 32128 catalog_manager.cc:5582] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:24.201191 32128 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]). Expected version 1 got 0
I20250623 08:04:24.201324 339 tablet_bootstrap.cc:654] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:24.203058 32121 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99ea7850738743deb7afa87ea2e40daa consensus state has no leader
I20250623 08:04:24.206421 344 ts_tablet_manager.cc:1918] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.206621 344 log.cc:1199] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/0f459b85abba44d288b0218c187f0098
I20250623 08:04:24.207096 344 ts_tablet_manager.cc:1939] T 0f459b85abba44d288b0218c187f0098 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:24.220360 32121 master_service.cc:948] ReplaceTablet: received request to replace tablet e6eb8420d9c849a5a751ed38fdac53a4 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.222352 32122 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d5617ea61e9d4010bf2ce97d61784515" } modify_external_catalogs: true
I20250623 08:04:24.222625 32122 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d5617ea61e9d4010bf2ce97d61784515" } modify_external_catalogs: true
W20250623 08:04:24.233744 32114 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 47470.4us
I20250623 08:04:24.234377 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b9d871507dcc46fc80f6e0ac739c6476 (table default.Table_d27db7e0fea34b5182353dee8ae23cac [id=816747bd5e004ae5964b7b885aa249fb]) successfully deleted
I20250623 08:04:24.235033 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 0f459b85abba44d288b0218c187f0098 (table default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]) successfully deleted
I20250623 08:04:24.236387 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 04eda92e14144733ab71a577dd3bd2b0 (DEFAULT_TABLE table=default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a [id=89a3fc1cb42f4902a69ea52945cedc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.236953 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04eda92e14144733ab71a577dd3bd2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.242352 362 raft_consensus.cc:491] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.242578 362 raft_consensus.cc:513] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.243455 362 leader_election.cc:290] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.244042 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b6f3fe1b10143aa9b54022546812498" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:24.244699 32314 leader_election.cc:343] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.245342 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b6f3fe1b10143aa9b54022546812498" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:24.245998 32314 leader_election.cc:343] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.246142 32314 leader_election.cc:304] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:24.246589 362 raft_consensus.cc:2747] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:24.250646 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_fae6ac5a256346149e8cf56ae9dbdf3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.251508 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fae6ac5a256346149e8cf56ae9dbdf3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.253137 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_e14bd6f5e20e42b1898bce238d4d986c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.253757 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e14bd6f5e20e42b1898bce238d4d986c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.255170 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_84c686e72347490ebc29749f8c9352f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.255810 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84c686e72347490ebc29749f8c9352f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.257723 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_284640472f6e45fba6c02c7e87c1ea05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.258311 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_284640472f6e45fba6c02c7e87c1ea05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.261240 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 04eda92e14144733ab71a577dd3bd2b0 (DEFAULT_TABLE table=default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a [id=89a3fc1cb42f4902a69ea52945cedc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.261222 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 04eda92e14144733ab71a577dd3bd2b0 (DEFAULT_TABLE table=default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a [id=89a3fc1cb42f4902a69ea52945cedc0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.261654 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04eda92e14144733ab71a577dd3bd2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.261655 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04eda92e14144733ab71a577dd3bd2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.261830 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 (DEFAULT_TABLE table=default.Table_8a1e4cf5c8a54882b63f1dfd10533986 [id=f9f9b316ab75478ca20caa217c01f3b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.265149 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 (DEFAULT_TABLE table=default.Table_8a1e4cf5c8a54882b63f1dfd10533986 [id=f9f9b316ab75478ca20caa217c01f3b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.265514 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 696f5d2797ef463bb72a2119501c6ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.265885 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d (DEFAULT_TABLE table=default.Table_c4287575a32b4847ba2379ef60074ba9 [id=abda89a30971466ea9a1f410d424ef58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.266191 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 696f5d2797ef463bb72a2119501c6ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.266932 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 (DEFAULT_TABLE table=default.Table_8a1e4cf5c8a54882b63f1dfd10533986 [id=f9f9b316ab75478ca20caa217c01f3b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.267269 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 696f5d2797ef463bb72a2119501c6ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.271024 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 4ac9f38877b24d619c280a1aaa0b7b08 (DEFAULT_TABLE table=default.Table_86a910b4ef524f2a84b98f25141cd248 [id=1f77f0d0a7fc4af9986e39dfee69bfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.271449 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ac9f38877b24d619c280a1aaa0b7b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.271590 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (DEFAULT_TABLE table=default.Table_a94b76793b7a49a687d0052141dd50c4 [id=f69e9adaf68a47f2a80d305c1936333b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.271929 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2414d24ab6c54f36bb1e554f7231a5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.275031 32495 tablet_service.cc:1468] Processing CreateTablet for tablet 228375592aee48a1aa21e1d16962b663 (DEFAULT_TABLE table=default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.275010 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (DEFAULT_TABLE table=default.Table_a94b76793b7a49a687d0052141dd50c4 [id=f69e9adaf68a47f2a80d305c1936333b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.275352 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2414d24ab6c54f36bb1e554f7231a5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.275352 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 228375592aee48a1aa21e1d16962b663. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.279021 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d (DEFAULT_TABLE table=default.Table_c4287575a32b4847ba2379ef60074ba9 [id=abda89a30971466ea9a1f410d424ef58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.279381 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.280773 32636 tablet_service.cc:1468] Processing CreateTablet for tablet ce1f3ef3690c4837b95b105733884f1f (DEFAULT_TABLE table=default.Table_5e8e88b2237f435595d619af47384b4e [id=dec4e6cfe22942698b2f2ae716f72e92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.281219 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1f3ef3690c4837b95b105733884f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.282897 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 228375592aee48a1aa21e1d16962b663 (DEFAULT_TABLE table=default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.283315 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 228375592aee48a1aa21e1d16962b663. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.286908 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c7d30889c024c84b6585f007504139e with delete_type TABLET_DATA_DELETED (replaced by tablet b9d871507dcc46fc80f6e0ac739c6476) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.287358 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 4ac9f38877b24d619c280a1aaa0b7b08 (DEFAULT_TABLE table=default.Table_86a910b4ef524f2a84b98f25141cd248 [id=1f77f0d0a7fc4af9986e39dfee69bfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.287707 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ac9f38877b24d619c280a1aaa0b7b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.291015 319 raft_consensus.cc:491] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.291250 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.291229 319 raft_consensus.cc:513] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.292105 319 leader_election.cc:290] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:24.294983 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99ea7850738743deb7afa87ea2e40daa" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:24.295352 32656 raft_consensus.cc:2466] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:24.295990 32453 leader_election.cc:304] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:24.296625 319 raft_consensus.cc:2802] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:24.296756 319 raft_consensus.cc:491] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:24.296854 319 raft_consensus.cc:3058] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:24.297559 32500 tablet_service.cc:1468] Processing CreateTablet for tablet ce1f3ef3690c4837b95b105733884f1f (DEFAULT_TABLE table=default.Table_5e8e88b2237f435595d619af47384b4e [id=dec4e6cfe22942698b2f2ae716f72e92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.298081 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1f3ef3690c4837b95b105733884f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.304010 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (DEFAULT_TABLE table=default.Table_f9ba2bae4aad4caa9c87e89dc65d3dc5 [id=8b20894a91c54a3eb08d5e1b4a42d269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.304420 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c01121f2e644d6aa6e8ec228bf75be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.304453 319 raft_consensus.cc:513] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:24.305727 32114 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 5c7d30889c024c84b6585f007504139e because the tablet was not found. No further retry: Not found: Tablet not found: 5c7d30889c024c84b6585f007504139e
I20250623 08:04:24.305738 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99ea7850738743deb7afa87ea2e40daa" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:24.305918 32656 raft_consensus.cc:3058] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:24.306082 32114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250623 08:04:24.309032 32656 raft_consensus.cc:2466] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:24.309695 32453 leader_election.cc:304] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:24.310286 453 raft_consensus.cc:2802] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:24.310900 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (DEFAULT_TABLE table=default.Table_f9ba2bae4aad4caa9c87e89dc65d3dc5 [id=8b20894a91c54a3eb08d5e1b4a42d269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.311360 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c01121f2e644d6aa6e8ec228bf75be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.316347 319 leader_election.cc:290] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:24.317399 453 raft_consensus.cc:695] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:24.317963 453 consensus_queue.cc:237] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.318504 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99ea7850738743deb7afa87ea2e40daa" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:24.319160 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 4ac9f38877b24d619c280a1aaa0b7b08 (DEFAULT_TABLE table=default.Table_86a910b4ef524f2a84b98f25141cd248 [id=1f77f0d0a7fc4af9986e39dfee69bfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.319929 32360 tablet_service.cc:1468] Processing CreateTablet for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d (DEFAULT_TABLE table=default.Table_c4287575a32b4847ba2379ef60074ba9 [id=abda89a30971466ea9a1f410d424ef58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.321452 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ac9f38877b24d619c280a1aaa0b7b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.326236 32123 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 0c77fd6e54904722acb2d1437c627ba8
I20250623 08:04:24.326853 32123 catalog_manager.cc:6652] ReplaceTablet: tablet 0c77fd6e54904722acb2d1437c627ba8 deleted and replaced by tablet 0861107ab4624a05bd7a4d24b420c502
I20250623 08:04:24.327283 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:24.327363 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c77fd6e54904722acb2d1437c627ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0861107ab4624a05bd7a4d24b420c502) from {username='slave'} at 127.0.0.1:55518
W20250623 08:04:24.327862 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.329095 32130 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 776b412b3f564a7581c6a5e1c84e8499
W20250623 08:04:24.329350 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 0c77fd6e54904722acb2d1437c627ba8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c77fd6e54904722acb2d1437c627ba8 already in progress: creating tablet
I20250623 08:04:24.329569 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_71cce1d280bf41b7bae2f47aeec9a39e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.330214 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71cce1d280bf41b7bae2f47aeec9a39e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.332259 32128 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a" } modify_external_catalogs: true
I20250623 08:04:24.332433 32128 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a" } modify_external_catalogs: true
I20250623 08:04:24.333628 32129 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]). Expected version 1 got 0
I20250623 08:04:24.335887 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_da3ed120aff049bc85b20e2200850ae3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.336380 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da3ed120aff049bc85b20e2200850ae3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.350203 32381 raft_consensus.cc:2466] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:24.355772 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (DEFAULT_TABLE table=default.Table_a94b76793b7a49a687d0052141dd50c4 [id=f69e9adaf68a47f2a80d305c1936333b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.357565 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.358389 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "99ea7850738743deb7afa87ea2e40daa" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:24.360229 32380 raft_consensus.cc:3058] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:24.362205 32356 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c77fd6e54904722acb2d1437c627ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0861107ab4624a05bd7a4d24b420c502) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.362147 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 228375592aee48a1aa21e1d16962b663 (DEFAULT_TABLE table=default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.362871 32357 tablet_service.cc:1468] Processing CreateTablet for tablet ce1f3ef3690c4837b95b105733884f1f (DEFAULT_TABLE table=default.Table_5e8e88b2237f435595d619af47384b4e [id=dec4e6cfe22942698b2f2ae716f72e92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.363093 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (DEFAULT_TABLE table=default.Table_f9ba2bae4aad4caa9c87e89dc65d3dc5 [id=8b20894a91c54a3eb08d5e1b4a42d269]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.364315 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 228375592aee48a1aa21e1d16962b663. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.367105 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2414d24ab6c54f36bb1e554f7231a5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.374262 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1f3ef3690c4837b95b105733884f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.374955 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c01121f2e644d6aa6e8ec228bf75be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.394452 32380 raft_consensus.cc:2466] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:24.422019 32121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet e6eb8420d9c849a5a751ed38fdac53a4
I20250623 08:04:24.422679 32121 catalog_manager.cc:6652] ReplaceTablet: tablet e6eb8420d9c849a5a751ed38fdac53a4 deleted and replaced by tablet 5fdc9e53da264a5ca1f70e1846c906f0
I20250623 08:04:24.424002 32121 catalog_manager.cc:5582] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:24.424613 32121 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_3cf27fe1edb94ead83092ffda8c5f7f1 [id=6cc9b03b74284f159ac54dd2dc10dc4b]). Expected version 3 got 0
I20250623 08:04:24.425673 32122 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 18b3287ab00043d99ad72c4e26dac4e1
I20250623 08:04:24.428910 32122 master_service.cc:948] ReplaceTablet: received request to replace tablet abf018986c24473688c8e5eec724f8ee from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.429581 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_86a7bf5fc79045769686a82168ac66da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.430367 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86a7bf5fc79045769686a82168ac66da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.431363 32127 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8a1e4cf5c8a54882b63f1dfd10533986" } modify_external_catalogs: true
I20250623 08:04:24.431607 32127 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8a1e4cf5c8a54882b63f1dfd10533986" } modify_external_catalogs: true
I20250623 08:04:24.433562 32125 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8cc5ec1f97f94a85bb54e58eb38760ca" } new_table_name: "default.Table_d895ba420734421e8b8f074848f4e389" modify_external_catalogs: true
I20250623 08:04:24.435473 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_8b1e8627d7a64cc394552d28e372c826"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.436215 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b1e8627d7a64cc394552d28e372c826 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.438266 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet e6eb8420d9c849a5a751ed38fdac53a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fdc9e53da264a5ca1f70e1846c906f0) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.438640 344 tablet_replica.cc:331] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:24.438916 344 raft_consensus.cc:2241] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.439173 344 raft_consensus.cc:2270] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.440104 344 ts_tablet_manager.cc:1905] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.449069 344 ts_tablet_manager.cc:1918] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.449334 344 log.cc:1199] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/e6eb8420d9c849a5a751ed38fdac53a4
I20250623 08:04:24.453241 344 ts_tablet_manager.cc:1939] T e6eb8420d9c849a5a751ed38fdac53a4 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:24.454808 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet e6eb8420d9c849a5a751ed38fdac53a4 (table default.Table_0e539ce89ea247e8b92fae1cc603ca88 [id=4e8632de2b4744ff9de198f82720a5bc]) successfully deleted
I20250623 08:04:24.467303 32356 tablet_service.cc:1515] Processing DeleteTablet for tablet abf018986c24473688c8e5eec724f8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.504369 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b6f3fe1b10143aa9b54022546812498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.508261 342 tablet_bootstrap.cc:492] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:24.508620 342 ts_tablet_manager.cc:1397] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.324s user 0.042s sys 0.076s
I20250623 08:04:24.509837 342 raft_consensus.cc:357] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.510177 342 raft_consensus.cc:383] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.510313 342 raft_consensus.cc:738] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.510659 342 consensus_queue.cc:260] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.511427 342 ts_tablet_manager.cc:1428] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:24.511929 342 tablet_bootstrap.cc:492] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:24.513769 342 tablet_bootstrap.cc:654] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.514237 366 raft_consensus.cc:491] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.514426 366 raft_consensus.cc:513] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.519439 366 leader_election.cc:290] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.519621 366 raft_consensus.cc:491] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.519778 366 raft_consensus.cc:513] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.520416 366 leader_election.cc:290] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.520896 32124 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_19e6011a07dd4ec08199244f47043017" } new_table_name: "default.Table_80649d448acb4b0b9c17043e97ca96c3" modify_external_catalogs: true
I20250623 08:04:24.521483 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c77fd6e54904722acb2d1437c627ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0861107ab4624a05bd7a4d24b420c502) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.522965 32123 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_25ef245b7f9344c0ad2e841fa78d0be7" } new_table_name: "default.Table_b7f7dd030adc46329a94b5de210eb2f9" modify_external_catalogs: true
I20250623 08:04:24.524600 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_97cf2d5cacff43bfbf001bd25c8f9676"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.525213 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97cf2d5cacff43bfbf001bd25c8f9676 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.527115 32121 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f9ba2bae4aad4caa9c87e89dc65d3dc5" } new_table_name: "default.Table_7e39ecbac4a8433e8ae98b0db22c30ea" modify_external_catalogs: true
I20250623 08:04:24.532235 32129 master_service.cc:948] ReplaceTablet: received request to replace tablet 9b6f3fe1b10143aa9b54022546812498 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.532761 32129 master_service.cc:948] ReplaceTablet: received request to replace tablet e6eb8420d9c849a5a751ed38fdac53a4 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.533345 32129 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_fe1953ea80584690bccd51204d81a314" } modify_external_catalogs: true
I20250623 08:04:24.533547 32129 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_fe1953ea80584690bccd51204d81a314" } modify_external_catalogs: true
I20250623 08:04:24.534494 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet e0c97148429046dc88213ff70a605c99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.534885 344 tablet_replica.cc:331] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:24.535133 344 raft_consensus.cc:2241] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.535316 344 raft_consensus.cc:2270] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.538146 344 ts_tablet_manager.cc:1905] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.546612 344 ts_tablet_manager.cc:1918] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.546775 344 log.cc:1199] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/e0c97148429046dc88213ff70a605c99
I20250623 08:04:24.547093 344 ts_tablet_manager.cc:1939] T e0c97148429046dc88213ff70a605c99 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:24.515655 32360 tablet_service.cc:1515] Processing DeleteTablet for tablet e6eb8420d9c849a5a751ed38fdac53a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fdc9e53da264a5ca1f70e1846c906f0) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.565321 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet e0c97148429046dc88213ff70a605c99 (table default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]) successfully deleted
I20250623 08:04:24.570912 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f459b85abba44d288b0218c187f0098 with delete_type TABLET_DATA_DELETED (replaced by tablet c1596e80225041099fb513ce477eae33) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.571166 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b5a56fe461641bea2bb269bf0564415 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:24.571208 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 0c77fd6e54904722acb2d1437c627ba8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c77fd6e54904722acb2d1437c627ba8 already in progress: creating tablet
I20250623 08:04:24.571388 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet a3bd7fa007544af69b3628911dca3a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.571600 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet e6eb8420d9c849a5a751ed38fdac53a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fdc9e53da264a5ca1f70e1846c906f0) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.571820 341 tablet_replica.cc:331] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:24.572002 341 raft_consensus.cc:2241] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.572182 341 raft_consensus.cc:2270] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.572973 341 ts_tablet_manager.cc:1905] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.576656 32656 raft_consensus.cc:3058] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:24.576817 32127 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 696f5d2797ef463bb72a2119501c6ef3
I20250623 08:04:24.580125 32520 raft_consensus.cc:1273] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Refusing update from remote peer 7351aa31800245659edc9837e27daffd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 08:04:24.583619 362 consensus_queue.cc:1035] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:24.584450 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.585018 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.585601 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet abf018986c24473688c8e5eec724f8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.588337 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a3c5a446184329b55fcc2cbf969831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.588961 342 tablet_bootstrap.cc:492] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:24.589108 342 ts_tablet_manager.cc:1397] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.012s
I20250623 08:04:24.589920 32656 raft_consensus.cc:1273] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Refusing update from remote peer 7351aa31800245659edc9837e27daffd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 08:04:24.593020 342 raft_consensus.cc:357] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.593230 342 raft_consensus.cc:383] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.593338 342 raft_consensus.cc:738] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.596127 342 consensus_queue.cc:260] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.598174 342 ts_tablet_manager.cc:1428] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20250623 08:04:24.598497 342 tablet_bootstrap.cc:492] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:24.600441 342 tablet_bootstrap.cc:654] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:24.602666 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 696f5d2797ef463bb72a2119501c6ef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 696f5d2797ef463bb72a2119501c6ef3 already in progress: creating tablet
I20250623 08:04:24.600466 341 ts_tablet_manager.cc:1918] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.605696 341 log.cc:1199] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/0f459b85abba44d288b0218c187f0098
I20250623 08:04:24.606443 341 ts_tablet_manager.cc:1939] T 0f459b85abba44d288b0218c187f0098 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:24.606891 362 consensus_queue.cc:1035] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250623 08:04:24.609802 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f9b870e59836470094157d5e22d85338" } new_table_name: "default.Table_8f909d419b67443f82c2c2d381c84a99" modify_external_catalogs: true
I20250623 08:04:24.618132 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 0f459b85abba44d288b0218c187f0098 (table default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]) successfully deleted
I20250623 08:04:24.620014 342 tablet_bootstrap.cc:492] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:24.620215 342 ts_tablet_manager.cc:1397] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
W20250623 08:04:24.620853 32125 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0ee5a4fe03848229524cc0105226d8c has no consensus state
I20250623 08:04:24.621122 342 raft_consensus.cc:357] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.621320 342 raft_consensus.cc:383] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.621388 342 raft_consensus.cc:738] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.621605 342 consensus_queue.cc:260] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.624456 341 tablet_replica.cc:331] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:24.624616 343 tablet_bootstrap.cc:492] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.624740 341 raft_consensus.cc:2241] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.624780 343 ts_tablet_manager.cc:1397] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.429s user 0.093s sys 0.132s
I20250623 08:04:24.624967 341 raft_consensus.cc:2270] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.625532 343 raft_consensus.cc:357] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.625764 343 raft_consensus.cc:383] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.625835 343 raft_consensus.cc:738] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.626048 343 consensus_queue.cc:260] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.626596 343 ts_tablet_manager.cc:1428] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:24.626772 32125 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c09ee28ff44c4c43835c2faa880a1c86" } modify_external_catalogs: true
I20250623 08:04:24.626895 343 tablet_bootstrap.cc:492] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.627017 32125 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c09ee28ff44c4c43835c2faa880a1c86" } modify_external_catalogs: true
I20250623 08:04:24.628511 343 tablet_bootstrap.cc:654] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.639626 340 ts_tablet_manager.cc:1918] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.642482 362 raft_consensus.cc:491] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.642654 362 raft_consensus.cc:513] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.643298 362 leader_election.cc:290] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.644235 339 tablet_bootstrap.cc:492] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:24.644158 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6eb8420d9c849a5a751ed38fdac53a4" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:24.644443 339 ts_tablet_manager.cc:1397] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.445s user 0.029s sys 0.111s
W20250623 08:04:24.644657 32314 leader_election.cc:343] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: e6eb8420d9c849a5a751ed38fdac53a4
W20250623 08:04:24.644868 32314 leader_election.cc:343] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.644999 32314 leader_election.cc:304] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:24.645376 362 raft_consensus.cc:2747] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:24.645561 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.645915 339 raft_consensus.cc:357] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.646181 339 raft_consensus.cc:383] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.646337 339 raft_consensus.cc:738] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.625527 341 ts_tablet_manager.cc:1905] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.646816 339 consensus_queue.cc:260] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.649466 340 log.cc:1199] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/3bf5243f44c043a0a1e9bc441fc91e70
I20250623 08:04:24.650259 340 ts_tablet_manager.cc:1939] T 3bf5243f44c043a0a1e9bc441fc91e70 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:24.651518 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 3bf5243f44c043a0a1e9bc441fc91e70 (table default.Table_13b0e15543044cd58b65937ca08d8df4 [id=adb0245ebf874699887e167d5b46d1d2]) successfully deleted
I20250623 08:04:24.652611 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6e29ab07df249748092a2f50a8c0273" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:24.653893 339 ts_tablet_manager.cc:1428] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250623 08:04:24.655587 339 tablet_bootstrap.cc:492] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:24.657970 339 tablet_bootstrap.cc:654] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:24.655172 32589 leader_election.cc:343] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: d3a3c5a446184329b55fcc2cbf969831
I20250623 08:04:24.671404 342 ts_tablet_manager.cc:1428] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.051s user 0.002s sys 0.001s
I20250623 08:04:24.671768 342 tablet_bootstrap.cc:492] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:24.658238 340 tablet_replica.cc:331] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:24.671880 453 raft_consensus.cc:491] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.672037 453 raft_consensus.cc:513] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.672122 340 raft_consensus.cc:2241] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.658655 481 tablet.cc:1722] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:24.659107 461 mvcc.cc:204] Tried to move back new op lower bound from 7170727381291782144 to 7170727379159486464. Current Snapshot: MvccSnapshot[applied={T|T < 7170727381291782144}]
I20250623 08:04:24.672652 453 leader_election.cc:290] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:24.672876 340 raft_consensus.cc:2270] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.673547 342 tablet_bootstrap.cc:654] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.676771 32380 raft_consensus.cc:1273] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Refusing update from remote peer 59f7c614254f42009306b13337f06576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 08:04:24.677965 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e0c11ec768a43a48c6dc0f3dd974b71" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:24.678700 453 consensus_queue.cc:1035] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:24.680121 32453 leader_election.cc:343] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.686829 32655 raft_consensus.cc:1273] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Refusing update from remote peer 59f7c614254f42009306b13337f06576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250623 08:04:24.687611 32124 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f02f36d6c1174a23a1f1043a431d0b6b has no consensus state
I20250623 08:04:24.687638 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e0c11ec768a43a48c6dc0f3dd974b71" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:24.687827 340 ts_tablet_manager.cc:1905] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:24.688706 32453 leader_election.cc:343] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.689110 32453 leader_election.cc:304] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:24.689406 453 consensus_queue.cc:1035] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:24.689855 453 raft_consensus.cc:2747] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:24.691285 482 tablet.cc:1722] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:24.700811 453 raft_consensus.cc:491] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.701058 453 raft_consensus.cc:513] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.701761 453 leader_election.cc:290] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:24.703799 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed5c81d9e56c4ee28546cedb9a579f31" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:24.704409 466 raft_consensus.cc:491] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.704552 466 raft_consensus.cc:513] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.704990 489 tablet.cc:1722] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:24.705276 466 leader_election.cc:290] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.705983 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abf018986c24473688c8e5eec724f8ee" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:24.706260 32381 raft_consensus.cc:2466] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:24.706351 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ed5c81d9e56c4ee28546cedb9a579f31" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:24.706923 32589 leader_election.cc:304] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:24.707268 466 raft_consensus.cc:2802] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:24.707379 466 raft_consensus.cc:491] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:24.707466 466 raft_consensus.cc:3058] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:24.710520 32130 master_service.cc:948] ReplaceTablet: received request to replace tablet 9b6f3fe1b10143aa9b54022546812498 from {username='slave'} at 127.0.0.1:50426
W20250623 08:04:24.711005 32121 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c01121f2e644d6aa6e8ec228bf75be9 consensus state has no leader
I20250623 08:04:24.711853 466 raft_consensus.cc:513] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.712539 466 leader_election.cc:290] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.713850 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6e29ab07df249748092a2f50a8c0273" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:24.714104 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abf018986c24473688c8e5eec724f8ee" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:24.714146 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abf018986c24473688c8e5eec724f8ee" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:24.714710 32453 leader_election.cc:343] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:24.715312 32453 leader_election.cc:343] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.715517 32453 leader_election.cc:304] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:24.716364 32589 leader_election.cc:304] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:24.717175 32589 leader_election.cc:343] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:24.717525 32589 leader_election.cc:343] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:24.717964 32589 leader_election.cc:343] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: d3a3c5a446184329b55fcc2cbf969831
I20250623 08:04:24.718097 32589 leader_election.cc:304] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:24.718370 366 raft_consensus.cc:2747] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:24.719074 319 raft_consensus.cc:2747] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:24.719165 343 tablet_bootstrap.cc:492] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:24.719412 343 ts_tablet_manager.cc:1397] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.000s
I20250623 08:04:24.720925 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "abf018986c24473688c8e5eec724f8ee" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:24.721113 32380 raft_consensus.cc:3058] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:24.722385 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_eff3794691a948cc9aa23546a7fe5457"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.723133 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eff3794691a948cc9aa23546a7fe5457 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.725227 32380 raft_consensus.cc:2466] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:24.725692 32121 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_10f030b47f8e4b31ba84115caef0203d" } modify_external_catalogs: true
I20250623 08:04:24.725914 32121 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_10f030b47f8e4b31ba84115caef0203d" } modify_external_catalogs: true
I20250623 08:04:24.729349 32126 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_25ef245b7f9344c0ad2e841fa78d0be7 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 1 got 0
W20250623 08:04:24.729940 32123 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7300f526d91f441bb164fe47f3a4f0ce consensus state has no leader
I20250623 08:04:24.730366 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_7a24313241f741ba965ed148a0380677"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.731041 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a24313241f741ba965ed148a0380677 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.733608 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_a3ef1d0ada5a4ef18db6aa59a43efad6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.734215 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3ef1d0ada5a4ef18db6aa59a43efad6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.720345 343 raft_consensus.cc:357] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.737511 340 ts_tablet_manager.cc:1918] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.737636 340 log.cc:1199] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b1db513f73d24116af299da4d267aabf
I20250623 08:04:24.739573 32122 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]). Expected version 1 got 0
I20250623 08:04:24.740095 32124 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]). Expected version 1 got 0
I20250623 08:04:24.740478 32122 master_service.cc:948] ReplaceTablet: received request to replace tablet 1e0c11ec768a43a48c6dc0f3dd974b71 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.740731 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet b1db513f73d24116af299da4d267aabf from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.741348 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_d000f5be42824b9fad2bdeb33231b5f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.741902 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d000f5be42824b9fad2bdeb33231b5f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.743961 340 ts_tablet_manager.cc:1939] T b1db513f73d24116af299da4d267aabf P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:24.743641 490 tablet.cc:1722] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:24.745601 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 (DEFAULT_TABLE table=default.Table_463abc7f8321496da24ccb06cba92d25 [id=7a569a9fe3bd4c608ddca48a734df1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.746018 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8f38a6de1e45e7bed4abdced684c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.753829 32499 tablet_service.cc:1468] Processing CreateTablet for tablet c6a97b34e386443fa19a9601621b5017 (DEFAULT_TABLE table=default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca [id=c330ef9f7f0549a7abc1eeab676ec11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.754253 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a97b34e386443fa19a9601621b5017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.756069 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 (DEFAULT_TABLE table=default.Table_463abc7f8321496da24ccb06cba92d25 [id=7a569a9fe3bd4c608ddca48a734df1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.756582 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8f38a6de1e45e7bed4abdced684c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.759052 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 726d80e05a344f27b5a289723cb5aed7 (DEFAULT_TABLE table=default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.759451 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726d80e05a344f27b5a289723cb5aed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.763069 32495 tablet_service.cc:1468] Processing CreateTablet for tablet 4ea81642050141a9be84f151a5785abc (DEFAULT_TABLE table=default.Table_ba70febc0949477ba25d45eafa4f7b43 [id=1475deb3ccfa49d5935900afa29dee6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.763458 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea81642050141a9be84f151a5785abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.767103 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 0311ce67989f49e8866234a88e5b4d09 (DEFAULT_TABLE table=default.Table_ba6fcef0af3d4e1cb50fa5dca6559223 [id=9d9e232100264e378516c06c61310750]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.767439 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0311ce67989f49e8866234a88e5b4d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.770152 342 tablet_bootstrap.cc:492] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:24.770289 342 ts_tablet_manager.cc:1397] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.099s user 0.001s sys 0.002s
I20250623 08:04:24.771840 32498 tablet_service.cc:1468] Processing CreateTablet for tablet d0ee5a4fe03848229524cc0105226d8c (DEFAULT_TABLE table=default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.771960 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 776b412b3f564a7581c6a5e1c84e8499 (DEFAULT_TABLE table=default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.772233 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0ee5a4fe03848229524cc0105226d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.772294 342 raft_consensus.cc:357] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.772516 342 raft_consensus.cc:383] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.772626 342 raft_consensus.cc:738] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.772927 342 consensus_queue.cc:260] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.776046 32359 tablet_service.cc:1468] Processing CreateTablet for tablet c6a97b34e386443fa19a9601621b5017 (DEFAULT_TABLE table=default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca [id=c330ef9f7f0549a7abc1eeab676ec11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.776463 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a97b34e386443fa19a9601621b5017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.777460 342 ts_tablet_manager.cc:1428] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250623 08:04:24.779045 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 726d80e05a344f27b5a289723cb5aed7 (DEFAULT_TABLE table=default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.779417 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726d80e05a344f27b5a289723cb5aed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.783077 32360 tablet_service.cc:1468] Processing CreateTablet for tablet ae7c2ac546114a819a543f1667fb7844 (DEFAULT_TABLE table=default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.783457 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7c2ac546114a819a543f1667fb7844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.787030 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 4ea81642050141a9be84f151a5785abc (DEFAULT_TABLE table=default.Table_ba70febc0949477ba25d45eafa4f7b43 [id=1475deb3ccfa49d5935900afa29dee6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.787379 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea81642050141a9be84f151a5785abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.787912 342 tablet_bootstrap.cc:492] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:24.791041 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 0311ce67989f49e8866234a88e5b4d09 (DEFAULT_TABLE table=default.Table_ba6fcef0af3d4e1cb50fa5dca6559223 [id=9d9e232100264e378516c06c61310750]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.791380 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0311ce67989f49e8866234a88e5b4d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.795024 32352 tablet_service.cc:1468] Processing CreateTablet for tablet d0ee5a4fe03848229524cc0105226d8c (DEFAULT_TABLE table=default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.795392 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0ee5a4fe03848229524cc0105226d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.799036 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 (DEFAULT_TABLE table=default.Table_c09ee28ff44c4c43835c2faa880a1c86 [id=371e56649fc6441c8aa52739de5aa077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.799364 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9918b15e48144ef0b8f3aedc2135be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.802878 32356 tablet_service.cc:1468] Processing CreateTablet for tablet f02f36d6c1174a23a1f1043a431d0b6b (DEFAULT_TABLE table=default.Table_80649d448acb4b0b9c17043e97ca96c3 [id=be88c6ecf20d4ab1b3757c27ce998f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.803252 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f02f36d6c1174a23a1f1043a431d0b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.806505 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776b412b3f564a7581c6a5e1c84e8499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.807714 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 776b412b3f564a7581c6a5e1c84e8499 (DEFAULT_TABLE table=default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.808050 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776b412b3f564a7581c6a5e1c84e8499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.813675 481 tablet.cc:1722] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250623 08:04:24.815222 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 consensus state has no leader
I20250623 08:04:24.816568 466 raft_consensus.cc:2747] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:24.816819 32589 leader_election.cc:304] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:24.817286 32127 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_3cf27fe1edb94ead83092ffda8c5f7f1 [id=6cc9b03b74284f159ac54dd2dc10dc4b]). Expected version 3 got 0
W20250623 08:04:24.821219 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 0c77fd6e54904722acb2d1437c627ba8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c77fd6e54904722acb2d1437c627ba8 already in progress: creating tablet
I20250623 08:04:24.821513 366 raft_consensus.cc:2802] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:24.823793 343 raft_consensus.cc:383] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.822474 32130 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_b7f7dd030adc46329a94b5de210eb2f9 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 2 got 0
I20250623 08:04:24.826089 473 mvcc.cc:204] Tried to move back new op lower bound from 7170727381578297344 to 7170727380258754560. Current Snapshot: MvccSnapshot[applied={T|T < 7170727381578297344}]
I20250623 08:04:24.826318 32130 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_3cf27fe1edb94ead83092ffda8c5f7f1 [id=6cc9b03b74284f159ac54dd2dc10dc4b]). Expected version 3 got 0
I20250623 08:04:24.823014 32129 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 04a3ae58d26142d58c4df4f52d7e3e2c
I20250623 08:04:24.827378 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 (DEFAULT_TABLE table=default.Table_463abc7f8321496da24ccb06cba92d25 [id=7a569a9fe3bd4c608ddca48a734df1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.828086 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_fd49e04fcac14f0194cc9fed910601fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:24.828799 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8f38a6de1e45e7bed4abdced684c92. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:24.828789 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd49e04fcac14f0194cc9fed910601fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.829421 366 raft_consensus.cc:695] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:24.823230 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 04eda92e14144733ab71a577dd3bd2b0
I20250623 08:04:24.831012 343 raft_consensus.cc:738] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.831265 343 consensus_queue.cc:260] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.833484 341 ts_tablet_manager.cc:1918] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.833685 32635 tablet_service.cc:1468] Processing CreateTablet for tablet c6a97b34e386443fa19a9601621b5017 (DEFAULT_TABLE table=default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca [id=c330ef9f7f0549a7abc1eeab676ec11d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.833791 32128 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_3cf27fe1edb94ead83092ffda8c5f7f1 [id=6cc9b03b74284f159ac54dd2dc10dc4b]). Expected version 3 got 0
I20250623 08:04:24.834017 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a97b34e386443fa19a9601621b5017. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.834403 339 tablet_bootstrap.cc:492] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:24.835211 339 ts_tablet_manager.cc:1397] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.179s user 0.003s sys 0.002s
I20250623 08:04:24.836021 340 tablet_replica.cc:331] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:24.837304 32356 tablet_service.cc:1515] Processing DeleteTablet for tablet 04a3ae58d26142d58c4df4f52d7e3e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.836549 366 consensus_queue.cc:237] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.839284 341 log.cc:1199] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/9b5a56fe461641bea2bb269bf0564415
I20250623 08:04:24.839329 340 raft_consensus.cc:2241] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.833113 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 726d80e05a344f27b5a289723cb5aed7 (DEFAULT_TABLE table=default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.839567 340 raft_consensus.cc:2270] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.840274 341 ts_tablet_manager.cc:1939] T 9b5a56fe461641bea2bb269bf0564415 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:24.840658 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_5174877154cc448c98758389a337e263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.841228 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5174877154cc448c98758389a337e263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.841866 341 tablet_replica.cc:331] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:24.842038 341 raft_consensus.cc:2241] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.843050 340 ts_tablet_manager.cc:1905] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.844691 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 04a3ae58d26142d58c4df4f52d7e3e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.845186 32356 tablet_service.cc:1515] Processing DeleteTablet for tablet 04eda92e14144733ab71a577dd3bd2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.845342 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 776b412b3f564a7581c6a5e1c84e8499 (DEFAULT_TABLE table=default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.846025 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 9b5a56fe461641bea2bb269bf0564415 (table default.Table_fac8d606d7454d3184af3a8d4c20ccc4 [id=d94a8f840a0c48a6bdfdca2ff5f0c5dc]) successfully deleted
I20250623 08:04:24.846097 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 4ea81642050141a9be84f151a5785abc (DEFAULT_TABLE table=default.Table_ba70febc0949477ba25d45eafa4f7b43 [id=1475deb3ccfa49d5935900afa29dee6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.846549 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726d80e05a344f27b5a289723cb5aed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.847023 341 raft_consensus.cc:2270] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.846407 32633 tablet_service.cc:1468] Processing CreateTablet for tablet ae7c2ac546114a819a543f1667fb7844 (DEFAULT_TABLE table=default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.850684 341 ts_tablet_manager.cc:1905] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.850828 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 776b412b3f564a7581c6a5e1c84e8499. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.852831 32625 tablet_service.cc:1515] Processing DeleteTablet for tablet 04eda92e14144733ab71a577dd3bd2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.853415 32632 tablet_service.cc:1468] Processing CreateTablet for tablet f02f36d6c1174a23a1f1043a431d0b6b (DEFAULT_TABLE table=default.Table_80649d448acb4b0b9c17043e97ca96c3 [id=be88c6ecf20d4ab1b3757c27ce998f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.859810 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 (DEFAULT_TABLE table=default.Table_c09ee28ff44c4c43835c2faa880a1c86 [id=371e56649fc6441c8aa52739de5aa077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.862473 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 (DEFAULT_TABLE table=default.Table_c09ee28ff44c4c43835c2faa880a1c86 [id=371e56649fc6441c8aa52739de5aa077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.862679 32492 tablet_service.cc:1468] Processing CreateTablet for tablet f02f36d6c1174a23a1f1043a431d0b6b (DEFAULT_TABLE table=default.Table_80649d448acb4b0b9c17043e97ca96c3 [id=be88c6ecf20d4ab1b3757c27ce998f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.862849 339 raft_consensus.cc:357] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.864005 490 tablet.cc:1717] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Skipping requested alter to schema version 3, tablet already version 3
I20250623 08:04:24.864109 32500 tablet_service.cc:1468] Processing CreateTablet for tablet ae7c2ac546114a819a543f1667fb7844 (DEFAULT_TABLE table=default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:24.866813 32116 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet f02f36d6c1174a23a1f1043a431d0b6b (table default.Table_80649d448acb4b0b9c17043e97ca96c3 [id=be88c6ecf20d4ab1b3757c27ce998f87], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f02f36d6c1174a23a1f1043a431d0b6b consensus state has no leader
I20250623 08:04:24.869133 339 raft_consensus.cc:383] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.869239 339 raft_consensus.cc:738] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.870309 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9918b15e48144ef0b8f3aedc2135be59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.870352 339 consensus_queue.cc:260] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.870998 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 04eda92e14144733ab71a577dd3bd2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.877626 339 ts_tablet_manager.cc:1428] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20250623 08:04:24.880847 469 raft_consensus.cc:491] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 08:04:24.881743 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 04eda92e14144733ab71a577dd3bd2b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04eda92e14144733ab71a577dd3bd2b0 already in progress: creating tablet
I20250623 08:04:24.882856 339 tablet_bootstrap.cc:492] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:24.880995 469 raft_consensus.cc:513] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.885104 469 leader_election.cc:290] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.885473 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_61ccd9db16cb4618874bff73413fd485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.886198 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61ccd9db16cb4618874bff73413fd485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.892468 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7c2ac546114a819a543f1667fb7844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.894814 453 raft_consensus.cc:491] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.894970 453 raft_consensus.cc:513] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.895545 453 leader_election.cc:290] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:24.895756 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 04a3ae58d26142d58c4df4f52d7e3e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.895682 339 tablet_bootstrap.cc:654] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.895920 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7300f526d91f441bb164fe47f3a4f0ce" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:24.898897 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f02f36d6c1174a23a1f1043a431d0b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.901257 340 ts_tablet_manager.cc:1918] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.901391 340 log.cc:1199] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/abf018986c24473688c8e5eec724f8ee
I20250623 08:04:24.901728 340 ts_tablet_manager.cc:1939] T abf018986c24473688c8e5eec724f8ee P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:24.902668 340 tablet_replica.cc:331] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:24.903077 32125 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 9918b15e48144ef0b8f3aedc2135be59
W20250623 08:04:24.903388 32314 leader_election.cc:343] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.903849 32125 master_service.cc:948] ReplaceTablet: received request to replace tablet b1db513f73d24116af299da4d267aabf from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.904404 32125 master_service.cc:948] ReplaceTablet: received request to replace tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:24.904855 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 1e0c11ec768a43a48c6dc0f3dd974b71
I20250623 08:04:24.905359 32122 catalog_manager.cc:6652] ReplaceTablet: tablet 1e0c11ec768a43a48c6dc0f3dd974b71 deleted and replaced by tablet 5629d11c44e241d598f3e519b2fe1a44
I20250623 08:04:24.905682 32122 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_eefa8d393087436793f03c22b76ab152" } modify_external_catalogs: true
I20250623 08:04:24.905889 32122 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_eefa8d393087436793f03c22b76ab152" } modify_external_catalogs: true
W20250623 08:04:24.907454 32453 leader_election.cc:343] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: b1db513f73d24116af299da4d267aabf
I20250623 08:04:24.907776 32124 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_97cf2d5cacff43bfbf001bd25c8f9676" } new_table_name: "default.Table_dfc3da325f6849bfba88293a5eabb3e4" modify_external_catalogs: true
I20250623 08:04:24.909179 32121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 228375592aee48a1aa21e1d16962b663
I20250623 08:04:24.909893 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_9b48914a8da94066afd2e04c5e639a95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.910567 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b48914a8da94066afd2e04c5e639a95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.912402 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_cd24de3acd89483480238a058a33e72c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.912858 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd24de3acd89483480238a058a33e72c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.918701 468 mvcc.cc:204] Tried to move back new op lower bound from 7170727382049832960 to 7170727380258754560. Current Snapshot: MvccSnapshot[applied={T|T < 7170727382049832960}]
I20250623 08:04:24.919566 32129 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_eff3794691a948cc9aa23546a7fe5457" } new_table_name: "default.Table_741648da5a6843bb89d2a36c8415f231" modify_external_catalogs: true
I20250623 08:04:24.921746 32127 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296] alter complete (version 1)
I20250623 08:04:24.922572 342 tablet_bootstrap.cc:654] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.923187 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_11b641dab5dd44b5b0ef1a309139d203" } new_table_name: "default.Table_d1cfca29bfb948dbb04297c66d9d2640" modify_external_catalogs: true
I20250623 08:04:24.925562 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_579c3f68f43e43b790fc83f17ad79191"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:24.926184 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_579c3f68f43e43b790fc83f17ad79191 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:24.926892 489 tablet.cc:1717] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Skipping requested alter to schema version 3, tablet already version 3
I20250623 08:04:24.932159 32626 tablet_service.cc:1468] Processing CreateTablet for tablet d0ee5a4fe03848229524cc0105226d8c (DEFAULT_TABLE table=default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.932847 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7c2ac546114a819a543f1667fb7844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.933405 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 0311ce67989f49e8866234a88e5b4d09 (DEFAULT_TABLE table=default.Table_ba6fcef0af3d4e1cb50fa5dca6559223 [id=9d9e232100264e378516c06c61310750]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:24.933995 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1db513f73d24116af299da4d267aabf" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:24.935076 466 raft_consensus.cc:491] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.935550 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7300f526d91f441bb164fe47f3a4f0ce" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:24.935714 340 raft_consensus.cc:2241] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.935911 340 raft_consensus.cc:2270] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.936472 340 ts_tablet_manager.cc:1905] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:24.936532 32314 leader_election.cc:343] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.936640 32314 leader_election.cc:304] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:24.937039 469 raft_consensus.cc:2747] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:24.937453 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0311ce67989f49e8866234a88e5b4d09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.937497 343 ts_tablet_manager.cc:1428] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.113s user 0.001s sys 0.001s
I20250623 08:04:24.938025 343 tablet_bootstrap.cc:492] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:24.938311 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0ee5a4fe03848229524cc0105226d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.938951 32130 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_80649d448acb4b0b9c17043e97ca96c3" } new_table_name: "default.Table_75451c4863a34537a328617c53d46cae" modify_external_catalogs: true
I20250623 08:04:24.939520 466 raft_consensus.cc:513] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.940989 466 leader_election.cc:290] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:24.942313 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f02f36d6c1174a23a1f1043a431d0b6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.944016 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0c97148429046dc88213ff70a605c99" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:24.946383 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9918b15e48144ef0b8f3aedc2135be59. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:24.956926 32453 leader_election.cc:343] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:24.957361 32453 leader_election.cc:304] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:24.957983 453 raft_consensus.cc:2747] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:24.959615 32114 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 60008.3us
I20250623 08:04:24.960584 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.960953 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.962121 32625 tablet_service.cc:1515] Processing DeleteTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.961617 481 tablet.cc:1717] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Skipping requested alter to schema version 3, tablet already version 3
I20250623 08:04:24.964146 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e0c11ec768a43a48c6dc0f3dd974b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 5629d11c44e241d598f3e519b2fe1a44) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.965142 32028 master_runner.cc:186] Error getting master registration for 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111),
I20250623 08:04:24.965317 32028 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250623 08:04:24.965438 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 228375592aee48a1aa21e1d16962b663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:24.967988 339 tablet_bootstrap.cc:492] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:24.968200 339 ts_tablet_manager.cc:1397] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.003s
I20250623 08:04:24.969024 339 raft_consensus.cc:357] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.969393 339 raft_consensus.cc:383] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.969532 339 raft_consensus.cc:738] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.969825 339 consensus_queue.cc:260] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.971197 339 ts_tablet_manager.cc:1428] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:24.971647 339 tablet_bootstrap.cc:492] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:24.974200 481 tablet.cc:1717] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Skipping requested alter to schema version 3, tablet already version 3
W20250623 08:04:24.974346 32589 leader_election.cc:343] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:24.974653 32589 leader_election.cc:343] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: e0c97148429046dc88213ff70a605c99
I20250623 08:04:24.975219 32589 leader_election.cc:304] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:24.976259 32625 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e0c11ec768a43a48c6dc0f3dd974b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 5629d11c44e241d598f3e519b2fe1a44) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.977126 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 228375592aee48a1aa21e1d16962b663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.978559 343 tablet_bootstrap.cc:654] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:24.981163 32624 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b6f3fe1b10143aa9b54022546812498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:24.981786 466 raft_consensus.cc:2747] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:24.988847 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea81642050141a9be84f151a5785abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:24.989598 340 ts_tablet_manager.cc:1918] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:24.989789 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e0c11ec768a43a48c6dc0f3dd974b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 5629d11c44e241d598f3e519b2fe1a44) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:24.990729 344 tablet_replica.cc:331] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:24.990953 344 raft_consensus.cc:2241] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:24.991511 344 raft_consensus.cc:2270] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:24.991777 342 tablet_bootstrap.cc:492] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:24.991928 342 ts_tablet_manager.cc:1397] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.204s user 0.007s sys 0.001s
I20250623 08:04:24.992303 344 ts_tablet_manager.cc:1905] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:24.992698 469 raft_consensus.cc:491] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:24.992709 342 raft_consensus.cc:357] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.992956 342 raft_consensus.cc:383] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:24.992890 469 raft_consensus.cc:513] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:24.993036 342 raft_consensus.cc:738] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:24.993275 342 consensus_queue.cc:260] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:24.993711 469 leader_election.cc:290] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:24.993767 342 ts_tablet_manager.cc:1428] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:24.994782 342 tablet_bootstrap.cc:492] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:24.995051 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet b1db513f73d24116af299da4d267aabf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
W20250623 08:04:24.995386 32314 leader_election.cc:343] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: e0c97148429046dc88213ff70a605c99
I20250623 08:04:25.000236 344 ts_tablet_manager.cc:1918] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.000340 339 tablet_bootstrap.cc:654] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.000486 344 log.cc:1199] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/1e0c11ec768a43a48c6dc0f3dd974b71
I20250623 08:04:25.000907 344 ts_tablet_manager.cc:1939] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:25.003057 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 04a3ae58d26142d58c4df4f52d7e3e2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04a3ae58d26142d58c4df4f52d7e3e2c already in progress: creating tablet
I20250623 08:04:25.003439 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0c97148429046dc88213ff70a605c99" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:25.003855 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 9918b15e48144ef0b8f3aedc2135be59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9918b15e48144ef0b8f3aedc2135be59 already in progress: creating tablet
I20250623 08:04:25.004102 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 1e0c11ec768a43a48c6dc0f3dd974b71 (table default.Table_11b641dab5dd44b5b0ef1a309139d203 [id=bb849e86d7084959830c68207fb99061]) successfully deleted
I20250623 08:04:25.004336 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet abf018986c24473688c8e5eec724f8ee (table default.Table_1ef4fa3350714732a2ecec291c255ed6 [id=affccd38bcd54f73af82d0ceeba931f8]) successfully deleted
I20250623 08:04:25.004601 32096 master_runner.cc:186] Error getting master registration for 127.31.69.60:39691: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111),
I20250623 08:04:25.004699 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet ed5c81d9e56c4ee28546cedb9a579f31 with delete_type TABLET_DATA_DELETED (replaced by tablet 04a3ae58d26142d58c4df4f52d7e3e2c) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.004804 32096 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250623 08:04:25.005839 32655 raft_consensus.cc:2466] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:25.006294 32314 leader_election.cc:304] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:25.006712 469 raft_consensus.cc:2802] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:25.006866 469 raft_consensus.cc:491] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:25.006987 469 raft_consensus.cc:3058] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:25.007300 342 tablet_bootstrap.cc:654] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.010921 339 tablet_bootstrap.cc:492] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.011099 339 ts_tablet_manager.cc:1397] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.002s
I20250623 08:04:25.011394 469 raft_consensus.cc:513] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.012254 469 leader_election.cc:290] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:25.013598 32314 leader_election.cc:343] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: e0c97148429046dc88213ff70a605c99
I20250623 08:04:25.013749 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e0c97148429046dc88213ff70a605c99" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:25.014132 32655 raft_consensus.cc:3058] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:24.989719 340 log.cc:1199] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/9b6f3fe1b10143aa9b54022546812498
I20250623 08:04:25.015894 340 ts_tablet_manager.cc:1939] T 9b6f3fe1b10143aa9b54022546812498 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:25.018780 32124 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e22f9bae5f7248af99cec06ce6dbbd41 has no consensus state
I20250623 08:04:25.022125 32121 master_service.cc:948] ReplaceTablet: received request to replace tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.022416 32655 raft_consensus.cc:2466] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:25.022360 339 raft_consensus.cc:357] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.022571 339 raft_consensus.cc:383] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.022645 339 raft_consensus.cc:738] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:25.022658 32129 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4221d7500d1458db75926d7b274a654 has no consensus state
I20250623 08:04:25.022902 32129 master_service.cc:948] ReplaceTablet: received request to replace tablet 726d80e05a344f27b5a289723cb5aed7 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.022938 339 consensus_queue.cc:260] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:25.023454 32130 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f02f36d6c1174a23a1f1043a431d0b6b consensus state has no leader
I20250623 08:04:25.023478 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet e90151394fb842498f4e780acab1f1aa
I20250623 08:04:25.025900 340 tablet_replica.cc:331] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:25.026077 340 raft_consensus.cc:2241] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.028079 340 raft_consensus.cc:2270] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.028611 340 ts_tablet_manager.cc:1905] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.031112 339 ts_tablet_manager.cc:1428] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250623 08:04:25.031430 339 tablet_bootstrap.cc:492] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.033735 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_11477a913b4b4b20ba2cbdd49dab0cca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:25.035483 32123 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_0e539ce89ea247e8b92fae1cc603ca88" } new_table_name: "default.Table_0242542a86ba40a28474c9832cd8ed3c" modify_external_catalogs: true
I20250623 08:04:25.036953 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 228375592aee48a1aa21e1d16962b663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.037999 32130 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_0b402e459582438db7f3ad072b276ee0" } modify_external_catalogs: true
I20250623 08:04:25.038200 32130 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_0b402e459582438db7f3ad072b276ee0" } modify_external_catalogs: true
I20250623 08:04:25.038986 32128 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_3cf27fe1edb94ead83092ffda8c5f7f1" } modify_external_catalogs: true
I20250623 08:04:25.039170 32128 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_3cf27fe1edb94ead83092ffda8c5f7f1" } modify_external_catalogs: true
W20250623 08:04:25.039671 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11477a913b4b4b20ba2cbdd49dab0cca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.044390 32314 leader_election.cc:304] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:25.045450 469 raft_consensus.cc:2802] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:25.045689 469 raft_consensus.cc:695] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:25.046145 469 consensus_queue.cc:237] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.055953 482 tablet.cc:1722] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:25.060678 344 tablet_replica.cc:331] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.061028 344 raft_consensus.cc:2241] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.061308 344 raft_consensus.cc:2270] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.061915 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 9b6f3fe1b10143aa9b54022546812498 (table default.Table_d2bb692b2c70474ea4b997cde4fe161a [id=33784d2b4a194d38b56b4881313dc03e]) successfully deleted
I20250623 08:04:25.062338 344 ts_tablet_manager.cc:1905] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.044646 339 tablet_bootstrap.cc:654] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.065910 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet e90151394fb842498f4e780acab1f1aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.065940 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet e90151394fb842498f4e780acab1f1aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.074375 340 ts_tablet_manager.cc:1918] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.074554 340 log.cc:1199] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/e6eb8420d9c849a5a751ed38fdac53a4
I20250623 08:04:25.075372 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet e90151394fb842498f4e780acab1f1aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.075426 339 tablet_bootstrap.cc:492] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.075619 339 ts_tablet_manager.cc:1397] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250623 08:04:25.076715 339 raft_consensus.cc:357] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.076958 339 raft_consensus.cc:383] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.077075 339 raft_consensus.cc:738] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.077392 339 consensus_queue.cc:260] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:25.079818 32123 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5fdc9e53da264a5ca1f70e1846c906f0 has no consensus state
I20250623 08:04:25.080228 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_4acf3aff222e477a885bc37d00623ddf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.080801 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4acf3aff222e477a885bc37d00623ddf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.085083 32124 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: default.Table_3cf27fe1edb94ead83092ffda8c5f7f1 [id=6cc9b03b74284f159ac54dd2dc10dc4b] alter complete (version 3)
I20250623 08:04:25.085333 340 ts_tablet_manager.cc:1939] T e6eb8420d9c849a5a751ed38fdac53a4 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:25.085844 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5629d11c44e241d598f3e519b2fe1a44 has no consensus state
I20250623 08:04:25.090447 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_6acaeaf59d9540dcab214d93b801a48e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.090999 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6acaeaf59d9540dcab214d93b801a48e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.095319 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet abf018986c24473688c8e5eec724f8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.096544 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 4fe30079045d437aa61bb5bfbfb96133 (DEFAULT_TABLE table=default.Table_86a7bf5fc79045769686a82168ac66da [id=6389457a4f384f9c9a69146391741b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.097031 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe30079045d437aa61bb5bfbfb96133. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.103736 32360 tablet_service.cc:1468] Processing CreateTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f (DEFAULT_TABLE table=default.Table_284640472f6e45fba6c02c7e87c1ea05 [id=bbf8e66e745141348c56d1cea5716898]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.103739 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 966a159e212040109e2842c15b69fb71 (DEFAULT_TABLE table=default.Table_84c686e72347490ebc29749f8c9352f9 [id=27884529aa7042ffb7eb7812892c6305]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.104102 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 966a159e212040109e2842c15b69fb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.104102 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ab9f77a81a4490abb0d2c0c69cd90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.107017 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 966a159e212040109e2842c15b69fb71 (DEFAULT_TABLE table=default.Table_84c686e72347490ebc29749f8c9352f9 [id=27884529aa7042ffb7eb7812892c6305]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.107369 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 966a159e212040109e2842c15b69fb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.108069 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d (DEFAULT_TABLE table=default.Table_fae6ac5a256346149e8cf56ae9dbdf3b [id=3c89160b4fb54f649fd3d222f6a6699d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.108565 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da84969a6464b5caeaf47af0cbaaa4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.111034 32497 tablet_service.cc:1468] Processing CreateTablet for tablet b980c746a3c042bfae90aa076acbb1df (DEFAULT_TABLE table=default.Table_71cce1d280bf41b7bae2f47aeec9a39e [id=09707632913546819a7f78d5d6bc5b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.111037 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 5e4678ca1c8f498c83c6bc2553882120 (DEFAULT_TABLE table=default.Table_e14bd6f5e20e42b1898bce238d4d986c [id=3a0c95fd4bd84b80ad1f5951cbef92e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.111227 339 ts_tablet_manager.cc:1428] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20250623 08:04:25.111415 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b980c746a3c042bfae90aa076acbb1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.112607 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d (DEFAULT_TABLE table=default.Table_fae6ac5a256346149e8cf56ae9dbdf3b [id=3c89160b4fb54f649fd3d222f6a6699d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.112942 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da84969a6464b5caeaf47af0cbaaa4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.114995 32491 tablet_service.cc:1468] Processing CreateTablet for tablet f0117cbb09dd4067836266c9d6a32adf (DEFAULT_TABLE table=default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.115015 32357 tablet_service.cc:1468] Processing CreateTablet for tablet b980c746a3c042bfae90aa076acbb1df (DEFAULT_TABLE table=default.Table_71cce1d280bf41b7bae2f47aeec9a39e [id=09707632913546819a7f78d5d6bc5b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.115319 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0117cbb09dd4067836266c9d6a32adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.115317 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b980c746a3c042bfae90aa076acbb1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.115932 339 tablet_bootstrap.cc:492] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.118225 339 tablet_bootstrap.cc:654] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.119022 32356 tablet_service.cc:1468] Processing CreateTablet for tablet f0117cbb09dd4067836266c9d6a32adf (DEFAULT_TABLE table=default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.119439 32495 tablet_service.cc:1468] Processing CreateTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 (DEFAULT_TABLE table=default.Table_dfc3da325f6849bfba88293a5eabb3e4 [id=ad93708e20ae4ef087d9a9b028d89806]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.119825 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e22f9bae5f7248af99cec06ce6dbbd41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.120689 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e4678ca1c8f498c83c6bc2553882120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.123086 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 5fdc9e53da264a5ca1f70e1846c906f0 (DEFAULT_TABLE table=default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.123576 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fdc9e53da264a5ca1f70e1846c906f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.128851 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 0861107ab4624a05bd7a4d24b420c502 (DEFAULT_TABLE table=default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.129308 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0861107ab4624a05bd7a4d24b420c502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.136209 32122 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_be622804eb114901b245383926c3afec" } modify_external_catalogs: true
I20250623 08:04:25.136443 32122 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_be622804eb114901b245383926c3afec" } modify_external_catalogs: true
I20250623 08:04:25.138089 32124 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a3ef1d0ada5a4ef18db6aa59a43efad6" } modify_external_catalogs: true
I20250623 08:04:25.138325 32124 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a3ef1d0ada5a4ef18db6aa59a43efad6" } modify_external_catalogs: true
I20250623 08:04:25.140458 32494 tablet_service.cc:1468] Processing CreateTablet for tablet effd4fff43fe4d4fb36458986e1dfce1 (DEFAULT_TABLE table=default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac [id=793edd0cb585499197d2f0e57a83be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.140918 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet effd4fff43fe4d4fb36458986e1dfce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.143188 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 1e47b10f46b8402e94354ebca573c194 (DEFAULT_TABLE table=default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.143616 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e47b10f46b8402e94354ebca573c194. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.144024 32129 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 726d80e05a344f27b5a289723cb5aed7
I20250623 08:04:25.144562 32129 catalog_manager.cc:6652] ReplaceTablet: tablet 726d80e05a344f27b5a289723cb5aed7 deleted and replaced by tablet 432d1642c0e44119b5bc05858463a136
I20250623 08:04:25.145004 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_5e1ad64ccae54db8bf7d5f8ebcfefb34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.145563 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1ad64ccae54db8bf7d5f8ebcfefb34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.145740 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_9788bd7b3d2848368b935832a2cffaed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.146275 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9788bd7b3d2848368b935832a2cffaed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.146854 32121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 7fea0b5aa0c1403fb3ef3ab9ab040035
I20250623 08:04:25.146961 32125 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d
I20250623 08:04:25.147251 32121 catalog_manager.cc:6652] ReplaceTablet: tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 deleted and replaced by tablet d02f74c24a174b18b08a870a4e47fefb
I20250623 08:04:25.147321 32125 catalog_manager.cc:6652] ReplaceTablet: tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d deleted and replaced by tablet c7fcf583064f4538913d9157e93068dd
I20250623 08:04:25.147511 32125 master_service.cc:948] ReplaceTablet: received request to replace tablet 8d8f38a6de1e45e7bed4abdced684c92 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.147603 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_af3136dbadfb43c892a197cc3a5d684e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.148075 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af3136dbadfb43c892a197cc3a5d684e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.148679 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 99ea7850738743deb7afa87ea2e40daa
I20250623 08:04:25.148645 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b6f3fe1b10143aa9b54022546812498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.149219 32128 master_service.cc:948] ReplaceTablet: received request to replace tablet 5ca4957928fc4630aa8d8ca6110d7598 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.150710 32130 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b
I20250623 08:04:25.156742 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d with delete_type TABLET_DATA_DELETED (replaced by tablet c7fcf583064f4538913d9157e93068dd) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.157142 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet 99ea7850738743deb7afa87ea2e40daa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.157272 32487 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 with delete_type TABLET_DATA_DELETED (replaced by tablet d02f74c24a174b18b08a870a4e47fefb) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.157516 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 726d80e05a344f27b5a289723cb5aed7 with delete_type TABLET_DATA_DELETED (replaced by tablet 432d1642c0e44119b5bc05858463a136) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.157753 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.159090 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f (DEFAULT_TABLE table=default.Table_284640472f6e45fba6c02c7e87c1ea05 [id=bbf8e66e745141348c56d1cea5716898]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.159574 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ab9f77a81a4490abb0d2c0c69cd90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.162994 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0117cbb09dd4067836266c9d6a32adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.173483 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 1e47b10f46b8402e94354ebca573c194 (DEFAULT_TABLE table=default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.176831 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 726d80e05a344f27b5a289723cb5aed7 with delete_type TABLET_DATA_DELETED (replaced by tablet 432d1642c0e44119b5bc05858463a136) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.177572 32361 tablet_service.cc:1468] Processing CreateTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 (DEFAULT_TABLE table=default.Table_dfc3da325f6849bfba88293a5eabb3e4 [id=ad93708e20ae4ef087d9a9b028d89806]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.178105 32349 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 with delete_type TABLET_DATA_DELETED (replaced by tablet d02f74c24a174b18b08a870a4e47fefb) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.182041 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e47b10f46b8402e94354ebca573c194. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.179028 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 5e4678ca1c8f498c83c6bc2553882120 (DEFAULT_TABLE table=default.Table_e14bd6f5e20e42b1898bce238d4d986c [id=3a0c95fd4bd84b80ad1f5951cbef92e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.182696 32358 tablet_service.cc:1468] Processing CreateTablet for tablet effd4fff43fe4d4fb36458986e1dfce1 (DEFAULT_TABLE table=default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac [id=793edd0cb585499197d2f0e57a83be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.183709 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 0861107ab4624a05bd7a4d24b420c502 (DEFAULT_TABLE table=default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.183408 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet e6eb8420d9c849a5a751ed38fdac53a4 (table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]) successfully deleted
I20250623 08:04:25.187389 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 4fe30079045d437aa61bb5bfbfb96133 (DEFAULT_TABLE table=default.Table_86a7bf5fc79045769686a82168ac66da [id=6389457a4f384f9c9a69146391741b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.187755 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe30079045d437aa61bb5bfbfb96133. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.188419 466 raft_consensus.cc:491] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:25.188676 466 raft_consensus.cc:513] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.189551 466 leader_election.cc:290] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:25.190935 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b6f3fe1b10143aa9b54022546812498" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:25.191238 32520 raft_consensus.cc:2466] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:25.192023 32589 leader_election.cc:304] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:25.192503 466 raft_consensus.cc:2802] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:25.192638 466 raft_consensus.cc:491] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:25.192763 466 raft_consensus.cc:3058] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:25.196975 466 raft_consensus.cc:513] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.198670 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e22f9bae5f7248af99cec06ce6dbbd41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.198998 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b6f3fe1b10143aa9b54022546812498" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:25.199244 32520 raft_consensus.cc:3058] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:25.200318 32349 tablet_service.cc:1515] Processing DeleteTablet for tablet 99ea7850738743deb7afa87ea2e40daa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.178951 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d with delete_type TABLET_DATA_DELETED (replaced by tablet c7fcf583064f4538913d9157e93068dd) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.202490 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.203313 342 tablet_bootstrap.cc:492] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:25.203482 342 ts_tablet_manager.cc:1397] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.209s user 0.004s sys 0.011s
I20250623 08:04:25.203549 32520 raft_consensus.cc:2466] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:25.204226 32589 leader_election.cc:304] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:25.204476 342 raft_consensus.cc:357] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.204681 342 raft_consensus.cc:383] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.204782 342 raft_consensus.cc:738] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.205044 342 consensus_queue.cc:260] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.205580 342 ts_tablet_manager.cc:1428] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250623 08:04:25.205881 342 tablet_bootstrap.cc:492] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:25.212146 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet e0c97148429046dc88213ff70a605c99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.212508 466 leader_election.cc:290] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:25.212653 342 tablet_bootstrap.cc:654] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.212795 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet b9d871507dcc46fc80f6e0ac739c6476 with delete_type TABLET_DATA_DELETED (replaced by tablet ca99c3b0cfec4aa481b620668b20947e) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.212770 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 5fdc9e53da264a5ca1f70e1846c906f0 (DEFAULT_TABLE table=default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.213059 466 raft_consensus.cc:2802] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:25.213394 466 raft_consensus.cc:695] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:25.213831 466 consensus_queue.cc:237] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.215091 340 tablet_replica.cc:331] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:25.219121 32347 tablet_service.cc:1515] Processing DeleteTablet for tablet c7385d86316f4a3ab68a77d18080ec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
W20250623 08:04:25.222491 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 696f5d2797ef463bb72a2119501c6ef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 696f5d2797ef463bb72a2119501c6ef3 already in progress: creating tablet
W20250623 08:04:25.224630 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 04a3ae58d26142d58c4df4f52d7e3e2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04a3ae58d26142d58c4df4f52d7e3e2c already in progress: creating tablet
W20250623 08:04:25.225044 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 04eda92e14144733ab71a577dd3bd2b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04eda92e14144733ab71a577dd3bd2b0 already in progress: creating tablet
I20250623 08:04:25.225200 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e4678ca1c8f498c83c6bc2553882120. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:25.225276 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 9918b15e48144ef0b8f3aedc2135be59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9918b15e48144ef0b8f3aedc2135be59 already in progress: creating tablet
I20250623 08:04:25.225445 340 raft_consensus.cc:2241] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.225646 340 raft_consensus.cc:2270] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.226059 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0861107ab4624a05bd7a4d24b420c502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.227456 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet effd4fff43fe4d4fb36458986e1dfce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.229593 340 ts_tablet_manager.cc:1905] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.232651 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fdc9e53da264a5ca1f70e1846c906f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.236728 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_f060288f2353427dafbc8caa87e5fdce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.237413 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f060288f2353427dafbc8caa87e5fdce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.238937 32130 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_86a910b4ef524f2a84b98f25141cd248" } new_table_name: "default.Table_2354480bbdc6474bb658478d129314d0" modify_external_catalogs: true
I20250623 08:04:25.240765 342 tablet_bootstrap.cc:492] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:25.240993 342 ts_tablet_manager.cc:1397] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.003s
I20250623 08:04:25.241189 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_741648da5a6843bb89d2a36c8415f231" } new_table_name: "default.Table_110816be0fae488f9cc26b302495009c" modify_external_catalogs: true
I20250623 08:04:25.241823 342 raft_consensus.cc:357] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.242094 342 raft_consensus.cc:383] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.242169 342 raft_consensus.cc:738] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.242445 342 consensus_queue.cc:260] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.243186 342 ts_tablet_manager.cc:1428] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:25.246871 342 tablet_bootstrap.cc:492] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:25.248797 342 tablet_bootstrap.cc:654] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.252136 32630 tablet_service.cc:1468] Processing CreateTablet for tablet f0117cbb09dd4067836266c9d6a32adf (DEFAULT_TABLE table=default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.252573 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0117cbb09dd4067836266c9d6a32adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.253398 32635 tablet_service.cc:1468] Processing CreateTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 (DEFAULT_TABLE table=default.Table_dfc3da325f6849bfba88293a5eabb3e4 [id=ad93708e20ae4ef087d9a9b028d89806]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.253737 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e22f9bae5f7248af99cec06ce6dbbd41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.254078 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 4fe30079045d437aa61bb5bfbfb96133 (DEFAULT_TABLE table=default.Table_86a7bf5fc79045769686a82168ac66da [id=6389457a4f384f9c9a69146391741b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.254382 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe30079045d437aa61bb5bfbfb96133. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.255122 32629 tablet_service.cc:1468] Processing CreateTablet for tablet b980c746a3c042bfae90aa076acbb1df (DEFAULT_TABLE table=default.Table_71cce1d280bf41b7bae2f47aeec9a39e [id=09707632913546819a7f78d5d6bc5b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.255528 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b980c746a3c042bfae90aa076acbb1df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.255568 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet d67a8f1f570741ebb886c4d136ca40d2
W20250623 08:04:25.257422 32589 leader_election.cc:343] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 9b6f3fe1b10143aa9b54022546812498
I20250623 08:04:25.257750 32122 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_ba6fcef0af3d4e1cb50fa5dca6559223" } new_table_name: "default.Table_e962b9945a5041039f7e33b98b653175" modify_external_catalogs: true
I20250623 08:04:25.258909 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d (DEFAULT_TABLE table=default.Table_fae6ac5a256346149e8cf56ae9dbdf3b [id=3c89160b4fb54f649fd3d222f6a6699d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.259056 32124 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet dd4152333cca4fa9a1461841a42d39f6
I20250623 08:04:25.259398 32124 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_cd24de3acd89483480238a058a33e72c" } modify_external_catalogs: true
I20250623 08:04:25.259574 32625 tablet_service.cc:1468] Processing CreateTablet for tablet 5e4678ca1c8f498c83c6bc2553882120 (DEFAULT_TABLE table=default.Table_e14bd6f5e20e42b1898bce238d4d986c [id=3a0c95fd4bd84b80ad1f5951cbef92e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.259614 32124 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_cd24de3acd89483480238a058a33e72c" } modify_external_catalogs: true
I20250623 08:04:25.260342 32624 tablet_service.cc:1468] Processing CreateTablet for tablet 1e47b10f46b8402e94354ebca573c194 (DEFAULT_TABLE table=default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.260496 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_4e377b569121400d82096403060f9cda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:25.260668 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 0861107ab4624a05bd7a4d24b420c502 (DEFAULT_TABLE table=default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.258111 344 ts_tablet_manager.cc:1918] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250623 08:04:25.261071 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e377b569121400d82096403060f9cda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.261204 344 log.cc:1199] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b1db513f73d24116af299da4d267aabf
I20250623 08:04:25.262483 344 ts_tablet_manager.cc:1939] T b1db513f73d24116af299da4d267aabf P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:25.263418 32125 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 8d8f38a6de1e45e7bed4abdced684c92
I20250623 08:04:25.263906 32125 catalog_manager.cc:6652] ReplaceTablet: tablet 8d8f38a6de1e45e7bed4abdced684c92 deleted and replaced by tablet 31a67b70cbc64bb3bfd5a34865109a01
I20250623 08:04:25.264235 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.264786 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.266083 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_1da3c8ca68084386b7f0d95a6f85927a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:25.266157 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 726d80e05a344f27b5a289723cb5aed7 with delete_type TABLET_DATA_DELETED (replaced by tablet 432d1642c0e44119b5bc05858463a136) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:25.266593 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1da3c8ca68084386b7f0d95a6f85927a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.266957 342 tablet_bootstrap.cc:492] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:25.267278 342 ts_tablet_manager.cc:1397] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20250623 08:04:25.267884 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 5ca4957928fc4630aa8d8ca6110d7598
I20250623 08:04:25.268288 32128 catalog_manager.cc:6652] ReplaceTablet: tablet 5ca4957928fc4630aa8d8ca6110d7598 deleted and replaced by tablet 67c44697ecab4dd9b43f544a602897ca
W20250623 08:04:25.268810 32589 leader_election.cc:343] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 9b6f3fe1b10143aa9b54022546812498
I20250623 08:04:25.268851 32632 tablet_service.cc:1468] Processing CreateTablet for tablet 966a159e212040109e2842c15b69fb71 (DEFAULT_TABLE table=default.Table_84c686e72347490ebc29749f8c9352f9 [id=27884529aa7042ffb7eb7812892c6305]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.269177 32621 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 with delete_type TABLET_DATA_DELETED (replaced by tablet d02f74c24a174b18b08a870a4e47fefb) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.269464 32623 tablet_service.cc:1468] Processing CreateTablet for tablet 5fdc9e53da264a5ca1f70e1846c906f0 (DEFAULT_TABLE table=default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.269807 32625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e4678ca1c8f498c83c6bc2553882120. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.270402 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 966a159e212040109e2842c15b69fb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.271060 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fdc9e53da264a5ca1f70e1846c906f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.271760 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0861107ab4624a05bd7a4d24b420c502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.272364 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e47b10f46b8402e94354ebca573c194. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.272918 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f (DEFAULT_TABLE table=default.Table_284640472f6e45fba6c02c7e87c1ea05 [id=bbf8e66e745141348c56d1cea5716898]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.273262 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ab9f77a81a4490abb0d2c0c69cd90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.274104 344 tablet_replica.cc:331] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.274344 344 raft_consensus.cc:2241] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.274492 32619 tablet_service.cc:1515] Processing DeleteTablet for tablet 99ea7850738743deb7afa87ea2e40daa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.275019 344 raft_consensus.cc:2270] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.275077 32633 tablet_service.cc:1468] Processing CreateTablet for tablet effd4fff43fe4d4fb36458986e1dfce1 (DEFAULT_TABLE table=default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac [id=793edd0cb585499197d2f0e57a83be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.275610 344 ts_tablet_manager.cc:1905] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.275674 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d with delete_type TABLET_DATA_DELETED (replaced by tablet c7fcf583064f4538913d9157e93068dd) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.277966 32619 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.274710 342 raft_consensus.cc:357] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.278852 342 raft_consensus.cc:383] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.278970 342 raft_consensus.cc:738] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.279210 342 consensus_queue.cc:260] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.280071 342 ts_tablet_manager.cc:1428] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250623 08:04:25.280416 342 tablet_bootstrap.cc:492] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:25.280725 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da84969a6464b5caeaf47af0cbaaa4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.282174 342 tablet_bootstrap.cc:654] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.284489 32128 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_5e1ad64ccae54db8bf7d5f8ebcfefb34" } new_table_name: "default.Table_b11754f05ec5489a8b6d9e1f5dbdd7c9" modify_external_catalogs: true
I20250623 08:04:25.291867 342 tablet_bootstrap.cc:492] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:25.292042 342 ts_tablet_manager.cc:1397] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250623 08:04:25.292779 342 raft_consensus.cc:357] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.292987 342 raft_consensus.cc:383] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.293052 342 raft_consensus.cc:738] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.293272 342 consensus_queue.cc:260] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:25.297137 32114 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 46385.5us
I20250623 08:04:25.300247 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet 125784661a7b4e4ba6a710b950e004f8 with delete_type TABLET_DATA_DELETED (replaced by tablet ae7c2ac546114a819a543f1667fb7844) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.300462 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet d67a8f1f570741ebb886c4d136ca40d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.300927 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 with delete_type TABLET_DATA_DELETED (replaced by tablet 31a67b70cbc64bb3bfd5a34865109a01) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.303136 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ca4957928fc4630aa8d8ca6110d7598 with delete_type TABLET_DATA_DELETED (replaced by tablet 67c44697ecab4dd9b43f544a602897ca) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.304248 342 ts_tablet_manager.cc:1428] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250623 08:04:25.305052 342 tablet_bootstrap.cc:492] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:25.306389 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet effd4fff43fe4d4fb36458986e1dfce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.306823 342 tablet_bootstrap.cc:654] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.310603 344 ts_tablet_manager.cc:1918] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.310807 344 log.cc:1199] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/ed5c81d9e56c4ee28546cedb9a579f31
I20250623 08:04:25.294739 492 tablet.cc:1717] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Skipping requested alter to schema version 3, tablet already version 3
I20250623 08:04:25.312362 344 ts_tablet_manager.cc:1939] T ed5c81d9e56c4ee28546cedb9a579f31 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:25.312947 492 tablet.cc:1717] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Skipping requested alter to schema version 3, tablet already version 3
I20250623 08:04:25.316924 344 tablet_replica.cc:331] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.317430 344 raft_consensus.cc:2241] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.318110 344 raft_consensus.cc:2270] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.319239 344 ts_tablet_manager.cc:1905] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.319284 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b1db513f73d24116af299da4d267aabf (table default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652]) successfully deleted
I20250623 08:04:25.319562 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet ed5c81d9e56c4ee28546cedb9a579f31 (table default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]) successfully deleted
W20250623 08:04:25.319804 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 228375592aee48a1aa21e1d16962b663 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 228375592aee48a1aa21e1d16962b663 already in progress: creating tablet
W20250623 08:04:25.322213 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet e90151394fb842498f4e780acab1f1aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e90151394fb842498f4e780acab1f1aa already in progress: creating tablet
W20250623 08:04:25.322661 32114 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 -> Peer 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691): Couldn't send request to peer 6644af0f4b2a4b998282b021ec1dc678. Status: Network error: Client connection negotiation failed: client connection to 127.31.69.60:39691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 08:04:25.327768 342 tablet_bootstrap.cc:492] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:25.328380 342 ts_tablet_manager.cc:1397] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250623 08:04:25.329304 342 raft_consensus.cc:357] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.331022 342 raft_consensus.cc:383] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.331286 342 raft_consensus.cc:738] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.331804 342 consensus_queue.cc:260] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.332029 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_63662f63088548e29e6db7281e66691e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:25.335645 342 ts_tablet_manager.cc:1428] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:25.335971 342 tablet_bootstrap.cc:492] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:25.339246 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63662f63088548e29e6db7281e66691e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:25.333297 32130 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ac9f38877b24d619c280a1aaa0b7b08 consensus state has no leader
W20250623 08:04:25.333556 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4221d7500d1458db75926d7b274a654 has no consensus state
W20250623 08:04:25.334414 32122 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0311ce67989f49e8866234a88e5b4d09 consensus state has no leader
W20250623 08:04:25.334682 32128 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 891fb836b1f74a9b9f561cc2ab63922c has no consensus state
I20250623 08:04:25.332907 32126 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_7a24313241f741ba965ed148a0380677" } new_table_name: "default.Table_24b73ca48b554c6698827e520b13b121" modify_external_catalogs: true
I20250623 08:04:25.344061 344 ts_tablet_manager.cc:1918] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.349021 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet e0c97148429046dc88213ff70a605c99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.349298 344 log.cc:1199] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/abf018986c24473688c8e5eec724f8ee
I20250623 08:04:25.349733 344 ts_tablet_manager.cc:1939] T abf018986c24473688c8e5eec724f8ee P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:25.345502 342 tablet_bootstrap.cc:654] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.353425 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_2b1942032adc43d78d1f9e83417b42eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.354014 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b1942032adc43d78d1f9e83417b42eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.357188 344 tablet_replica.cc:331] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.357460 344 raft_consensus.cc:2241] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.357849 344 raft_consensus.cc:2270] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.358798 344 ts_tablet_manager.cc:1905] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.360731 32127 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_47264042125847ac997923572c293a3e" } modify_external_catalogs: true
I20250623 08:04:25.360909 32127 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_47264042125847ac997923572c293a3e" } modify_external_catalogs: true
I20250623 08:04:25.361451 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_92ca14ac8b9647b0bd3b515217b30323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.361929 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92ca14ac8b9647b0bd3b515217b30323 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.363138 32125 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_b11754f05ec5489a8b6d9e1f5dbdd7c9" } modify_external_catalogs: true
I20250623 08:04:25.363299 32125 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_b11754f05ec5489a8b6d9e1f5dbdd7c9" } modify_external_catalogs: true
I20250623 08:04:25.364487 343 tablet_bootstrap.cc:492] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:25.364665 343 ts_tablet_manager.cc:1397] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.427s user 0.014s sys 0.015s
I20250623 08:04:25.365376 343 raft_consensus.cc:357] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.365552 343 raft_consensus.cc:383] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.365620 343 raft_consensus.cc:738] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.365883 343 consensus_queue.cc:260] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.366739 343 ts_tablet_manager.cc:1428] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:25.366817 453 raft_consensus.cc:491] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:25.366971 453 raft_consensus.cc:513] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.367357 32129 master_service.cc:948] ReplaceTablet: received request to replace tablet 1e47b10f46b8402e94354ebca573c194 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.367640 32121 master_service.cc:948] ReplaceTablet: received request to replace tablet 696f5d2797ef463bb72a2119501c6ef3 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.367650 453 leader_election.cc:290] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:25.368117 32121 master_service.cc:948] ReplaceTablet: received request to replace tablet c7385d86316f4a3ab68a77d18080ec20 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.372010 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a46c0a89fc014054a75ed198e824b5b6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:25.372447 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet abf018986c24473688c8e5eec724f8ee (table default.Table_1ef4fa3350714732a2ecec291c255ed6 [id=affccd38bcd54f73af82d0ceeba931f8]) successfully deleted
W20250623 08:04:25.372838 32453 leader_election.cc:343] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.379714 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet b1db513f73d24116af299da4d267aabf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.381191 344 ts_tablet_manager.cc:1918] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.381810 344 log.cc:1199] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9b6f3fe1b10143aa9b54022546812498
I20250623 08:04:25.382406 344 ts_tablet_manager.cc:1939] T 9b6f3fe1b10143aa9b54022546812498 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:25.298563 341 ts_tablet_manager.cc:1918] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.298635 32619 tablet_service.cc:1515] Processing DeleteTablet for tablet d67a8f1f570741ebb886c4d136ca40d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.301158 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 with delete_type TABLET_DATA_DELETED (replaced by tablet 31a67b70cbc64bb3bfd5a34865109a01) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.301566 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ca4957928fc4630aa8d8ca6110d7598 with delete_type TABLET_DATA_DELETED (replaced by tablet 67c44697ecab4dd9b43f544a602897ca) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.383385 341 log.cc:1199] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/a3bd7fa007544af69b3628911dca3a80
I20250623 08:04:25.383790 341 ts_tablet_manager.cc:1939] T a3bd7fa007544af69b3628911dca3a80 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:25.384719 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet a3bd7fa007544af69b3628911dca3a80 (table default.Table_0beadbd550664534bbf90d51ca7a4b0b [id=967a9b35621a401681cb78fb66402525]) successfully deleted
I20250623 08:04:25.384954 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 9b6f3fe1b10143aa9b54022546812498 (table default.Table_d2bb692b2c70474ea4b997cde4fe161a [id=33784d2b4a194d38b56b4881313dc03e]) successfully deleted
W20250623 08:04:25.385181 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet e6eb8420d9c849a5a751ed38fdac53a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6eb8420d9c849a5a751ed38fdac53a4 already in progress: creating tablet
I20250623 08:04:25.385250 341 tablet_replica.cc:331] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
W20250623 08:04:25.385461 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 696f5d2797ef463bb72a2119501c6ef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 696f5d2797ef463bb72a2119501c6ef3 already in progress: creating tablet
I20250623 08:04:25.385484 341 raft_consensus.cc:2241] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:25.385900 341 pending_rounds.cc:70] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:25.386085 341 pending_rounds.cc:77] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727382379438080 op_type: NO_OP noop_request { }
I20250623 08:04:25.386219 341 raft_consensus.cc:2887] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:25.386354 341 raft_consensus.cc:2270] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.388425 341 ts_tablet_manager.cc:1905] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:25.389868 32126 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 417a050eada4403ebbddff255fb6fb5a has no consensus state
I20250623 08:04:25.395802 32128 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_b7f7dd030adc46329a94b5de210eb2f9 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 2 got 0
I20250623 08:04:25.397269 343 tablet_bootstrap.cc:492] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:25.397619 344 tablet_replica.cc:331] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.397825 344 raft_consensus.cc:2241] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:25.398433 344 raft_consensus.cc:2270] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.404448 344 ts_tablet_manager.cc:1905] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.404987 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a46c0a89fc014054a75ed198e824b5b6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:25.405745 32453 leader_election.cc:343] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.405881 32453 leader_election.cc:304] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:25.406273 453 raft_consensus.cc:2747] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:25.407203 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d already in progress: creating tablet
I20250623 08:04:25.407483 469 raft_consensus.cc:491] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:25.407951 469 raft_consensus.cc:513] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.408099 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet c7385d86316f4a3ab68a77d18080ec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.408450 32624 tablet_service.cc:1468] Processing CreateTablet for tablet 175ee90133994154b7e4b58b0770db3b (DEFAULT_TABLE table=default.Table_11477a913b4b4b20ba2cbdd49dab0cca [id=426f83a7100640699646c6a02f20c6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.408572 469 leader_election.cc:290] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:25.408927 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 175ee90133994154b7e4b58b0770db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.409236 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet a46c0a89fc014054a75ed198e824b5b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 228375592aee48a1aa21e1d16962b663) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.409232 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ca4957928fc4630aa8d8ca6110d7598 with delete_type TABLET_DATA_DELETED (replaced by tablet 67c44697ecab4dd9b43f544a602897ca) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.409405 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7385d86316f4a3ab68a77d18080ec20" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:25.409662 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 175ee90133994154b7e4b58b0770db3b (DEFAULT_TABLE table=default.Table_11477a913b4b4b20ba2cbdd49dab0cca [id=426f83a7100640699646c6a02f20c6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.410022 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 175ee90133994154b7e4b58b0770db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.411094 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7385d86316f4a3ab68a77d18080ec20" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:25.411303 32520 raft_consensus.cc:2466] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
W20250623 08:04:25.412823 32314 leader_election.cc:343] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.413053 32314 leader_election.cc:304] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:25.415472 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d (DEFAULT_TABLE table=default.Table_cd24de3acd89483480238a058a33e72c [id=9b918ea1a6ca412d9db93bbb1215732f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.415678 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d (DEFAULT_TABLE table=default.Table_cd24de3acd89483480238a058a33e72c [id=9b918ea1a6ca412d9db93bbb1215732f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.415823 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2795cb14e6db4116a4c9ee539bd4e82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.416087 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2795cb14e6db4116a4c9ee539bd4e82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.418599 469 raft_consensus.cc:2802] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:25.418706 469 raft_consensus.cc:491] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:25.421886 469 raft_consensus.cc:3058] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:25.423295 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d (DEFAULT_TABLE table=default.Table_cd24de3acd89483480238a058a33e72c [id=9b918ea1a6ca412d9db93bbb1215732f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.426436 32356 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 with delete_type TABLET_DATA_DELETED (replaced by tablet 31a67b70cbc64bb3bfd5a34865109a01) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.426797 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa (DEFAULT_TABLE table=default.Table_9b48914a8da94066afd2e04c5e639a95 [id=73ffec23a77441678c2f9d76087ba5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.426992 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2795cb14e6db4116a4c9ee539bd4e82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.427016 32623 tablet_service.cc:1468] Processing CreateTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa (DEFAULT_TABLE table=default.Table_9b48914a8da94066afd2e04c5e639a95 [id=73ffec23a77441678c2f9d76087ba5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.427199 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.427457 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.431452 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 26f4a1b472e848ccb9a1ec71c1dd3516 (DEFAULT_TABLE table=default.Table_61ccd9db16cb4618874bff73413fd485 [id=8d6f6785bcf24b719419015368a05f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.432169 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 417a050eada4403ebbddff255fb6fb5a (DEFAULT_TABLE table=default.Table_24b73ca48b554c6698827e520b13b121 [id=2857dcb54e8b4109996cea363085a395]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.432469 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f4a1b472e848ccb9a1ec71c1dd3516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.432741 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417a050eada4403ebbddff255fb6fb5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.432749 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 (DEFAULT_TABLE table=default.Table_d1cfca29bfb948dbb04297c66d9d2640 [id=bb849e86d7084959830c68207fb99061]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.433274 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5629d11c44e241d598f3e519b2fe1a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.434010 32356 tablet_service.cc:1468] Processing CreateTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa (DEFAULT_TABLE table=default.Table_9b48914a8da94066afd2e04c5e639a95 [id=73ffec23a77441678c2f9d76087ba5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.434434 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.436044 469 raft_consensus.cc:513] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.437999 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 417a050eada4403ebbddff255fb6fb5a (DEFAULT_TABLE table=default.Table_24b73ca48b554c6698827e520b13b121 [id=2857dcb54e8b4109996cea363085a395]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.438344 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417a050eada4403ebbddff255fb6fb5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.439683 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 26f4a1b472e848ccb9a1ec71c1dd3516 (DEFAULT_TABLE table=default.Table_61ccd9db16cb4618874bff73413fd485 [id=8d6f6785bcf24b719419015368a05f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.440057 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f4a1b472e848ccb9a1ec71c1dd3516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.441490 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7385d86316f4a3ab68a77d18080ec20" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:25.441669 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 (DEFAULT_TABLE table=default.Table_d1cfca29bfb948dbb04297c66d9d2640 [id=bb849e86d7084959830c68207fb99061]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.442005 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5629d11c44e241d598f3e519b2fe1a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.442090 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 485538f7c63948818f20a25c02381519 (DEFAULT_TABLE table=default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.442516 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485538f7c63948818f20a25c02381519. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:25.443939 32314 leader_election.cc:343] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.444844 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7385d86316f4a3ab68a77d18080ec20" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:25.445019 32520 raft_consensus.cc:3058] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:25.447224 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 485538f7c63948818f20a25c02381519 (DEFAULT_TABLE table=default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.447542 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485538f7c63948818f20a25c02381519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.447544 32620 tablet_service.cc:1468] Processing CreateTablet for tablet f3c850021e914530bb106ab7166be7c9 (DEFAULT_TABLE table=default.Table_fd49e04fcac14f0194cc9fed910601fc [id=17b055700da64cc1b19f93dd5edf40d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.447861 32520 raft_consensus.cc:2466] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:25.448043 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c850021e914530bb106ab7166be7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.451318 32491 tablet_service.cc:1468] Processing CreateTablet for tablet f3c850021e914530bb106ab7166be7c9 (DEFAULT_TABLE table=default.Table_fd49e04fcac14f0194cc9fed910601fc [id=17b055700da64cc1b19f93dd5edf40d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.451660 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c850021e914530bb106ab7166be7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.452201 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 26f4a1b472e848ccb9a1ec71c1dd3516 (DEFAULT_TABLE table=default.Table_61ccd9db16cb4618874bff73413fd485 [id=8d6f6785bcf24b719419015368a05f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.452623 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f4a1b472e848ccb9a1ec71c1dd3516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.454017 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 417a050eada4403ebbddff255fb6fb5a (DEFAULT_TABLE table=default.Table_24b73ca48b554c6698827e520b13b121 [id=2857dcb54e8b4109996cea363085a395]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.463434 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 485538f7c63948818f20a25c02381519 (DEFAULT_TABLE table=default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.456003 32314 leader_election.cc:304] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:25.469955 32128 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca" } modify_external_catalogs: true
I20250623 08:04:25.470178 32128 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca" } modify_external_catalogs: true
I20250623 08:04:25.470719 32126 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f060288f2353427dafbc8caa87e5fdce" } modify_external_catalogs: true
I20250623 08:04:25.470952 32126 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f060288f2353427dafbc8caa87e5fdce" } modify_external_catalogs: true
I20250623 08:04:25.472438 32357 tablet_service.cc:1468] Processing CreateTablet for tablet f3c850021e914530bb106ab7166be7c9 (DEFAULT_TABLE table=default.Table_fd49e04fcac14f0194cc9fed910601fc [id=17b055700da64cc1b19f93dd5edf40d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.474635 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_7987e1a82b644bd69724bc61cd84445a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.475136 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7987e1a82b644bd69724bc61cd84445a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.479192 469 leader_election.cc:290] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:25.479538 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 (DEFAULT_TABLE table=default.Table_d1cfca29bfb948dbb04297c66d9d2640 [id=bb849e86d7084959830c68207fb99061]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.483233 32496 tablet_service.cc:1468] Processing CreateTablet for tablet d4221d7500d1458db75926d7b274a654 (DEFAULT_TABLE table=default.Table_110816be0fae488f9cc26b302495009c [id=6de8bb3add89407bb0d3e5685ed5b6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.483631 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4221d7500d1458db75926d7b274a654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.482153 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 175ee90133994154b7e4b58b0770db3b (DEFAULT_TABLE table=default.Table_11477a913b4b4b20ba2cbdd49dab0cca [id=426f83a7100640699646c6a02f20c6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.484144 362 raft_consensus.cc:2802] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:25.484292 362 raft_consensus.cc:695] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:25.489912 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485538f7c63948818f20a25c02381519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.490393 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet d67a8f1f570741ebb886c4d136ca40d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.494069 340 ts_tablet_manager.cc:1918] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.494457 340 log.cc:1199] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/1e0c11ec768a43a48c6dc0f3dd974b71
I20250623 08:04:25.495254 342 tablet_bootstrap.cc:492] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:25.495462 342 ts_tablet_manager.cc:1397] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.160s user 0.006s sys 0.009s
I20250623 08:04:25.496042 362 consensus_queue.cc:237] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.496526 342 raft_consensus.cc:357] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.496809 342 raft_consensus.cc:383] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.496896 342 raft_consensus.cc:738] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.497171 342 consensus_queue.cc:260] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.497782 342 ts_tablet_manager.cc:1428] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:25.499254 342 tablet_bootstrap.cc:492] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:25.500321 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5629d11c44e241d598f3e519b2fe1a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.501411 342 tablet_bootstrap.cc:654] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.505420 340 ts_tablet_manager.cc:1939] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:25.506513 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 279624b8bb834b95a4f08045968ebfb4 (DEFAULT_TABLE table=default.Table_d000f5be42824b9fad2bdeb33231b5f8 [id=d68afa3a431d4386b862bd4b47a5e1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.506513 32352 tablet_service.cc:1468] Processing CreateTablet for tablet d4221d7500d1458db75926d7b274a654 (DEFAULT_TABLE table=default.Table_110816be0fae488f9cc26b302495009c [id=6de8bb3add89407bb0d3e5685ed5b6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.507949 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 175ee90133994154b7e4b58b0770db3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.508600 344 ts_tablet_manager.cc:1918] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250623 08:04:25.508724 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4221d7500d1458db75926d7b274a654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.508741 344 log.cc:1199] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/99ea7850738743deb7afa87ea2e40daa
I20250623 08:04:25.509609 32492 tablet_service.cc:1468] Processing CreateTablet for tablet cde7df7d39a54114a4dcb28cfd1eb429 (DEFAULT_TABLE table=default.Table_5174877154cc448c98758389a337e263 [id=e25e0edebbac486785025ef14bcf5352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.510346 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 279624b8bb834b95a4f08045968ebfb4 (DEFAULT_TABLE table=default.Table_d000f5be42824b9fad2bdeb33231b5f8 [id=d68afa3a431d4386b862bd4b47a5e1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.511080 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 279624b8bb834b95a4f08045968ebfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.513634 32354 tablet_service.cc:1468] Processing CreateTablet for tablet cde7df7d39a54114a4dcb28cfd1eb429 (DEFAULT_TABLE table=default.Table_5174877154cc448c98758389a337e263 [id=e25e0edebbac486785025ef14bcf5352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.515384 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_e2502c251f3c4a7f8caa4faab352ebda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.515923 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2502c251f3c4a7f8caa4faab352ebda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.517208 32122 master_service.cc:948] ReplaceTablet: received request to replace tablet d0ee5a4fe03848229524cc0105226d8c from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.518478 32130 master_service.cc:948] ReplaceTablet: received request to replace tablet f0117cbb09dd4067836266c9d6a32adf from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.518596 344 ts_tablet_manager.cc:1939] T 99ea7850738743deb7afa87ea2e40daa P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:25.519766 32125 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 891fb836b1f74a9b9f561cc2ab63922c
I20250623 08:04:25.519846 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde7df7d39a54114a4dcb28cfd1eb429. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.520117 32129 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 1e47b10f46b8402e94354ebca573c194
I20250623 08:04:25.520673 32129 catalog_manager.cc:6652] ReplaceTablet: tablet 1e47b10f46b8402e94354ebca573c194 deleted and replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2
I20250623 08:04:25.521057 32127 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet b6ad6cd629c4423ca997efb517cc975b
I20250623 08:04:25.524562 32124 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 2795cb14e6db4116a4c9ee539bd4e82d
I20250623 08:04:25.526259 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 279624b8bb834b95a4f08045968ebfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.534093 344 tablet_replica.cc:331] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.534314 344 raft_consensus.cc:2241] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.534680 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e47b10f46b8402e94354ebca573c194 with delete_type TABLET_DATA_DELETED (replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.535748 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.535941 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ad6cd629c4423ca997efb517cc975b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.534147 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e47b10f46b8402e94354ebca573c194 with delete_type TABLET_DATA_DELETED (replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.536782 344 raft_consensus.cc:2270] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.535718 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ad6cd629c4423ca997efb517cc975b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.538058 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.538622 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet e90151394fb842498f4e780acab1f1aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.539192 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_3878732ceac64ef6bc3f0d170ffbe12a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:25.539381 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 1e0c11ec768a43a48c6dc0f3dd974b71 (table default.Table_d1cfca29bfb948dbb04297c66d9d2640 [id=bb849e86d7084959830c68207fb99061]) successfully deleted
I20250623 08:04:25.539700 344 ts_tablet_manager.cc:1905] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:25.539781 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3878732ceac64ef6bc3f0d170ffbe12a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:25.540365 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 228375592aee48a1aa21e1d16962b663 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 228375592aee48a1aa21e1d16962b663 already in progress: creating tablet
W20250623 08:04:25.540647 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet e90151394fb842498f4e780acab1f1aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e90151394fb842498f4e780acab1f1aa already in progress: creating tablet
I20250623 08:04:25.538635 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_fb91c1ae95e04cefba3aa6b9c00ecc12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.541672 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb91c1ae95e04cefba3aa6b9c00ecc12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.541944 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_f47e2bf1990949db8996199686e7034f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.542500 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f47e2bf1990949db8996199686e7034f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.546088 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_e6b4dee87da24acdbfd7d121769b4d86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.546669 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6b4dee87da24acdbfd7d121769b4d86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.546854 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde7df7d39a54114a4dcb28cfd1eb429. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.551386 340 tablet_replica.cc:331] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:25.551596 340 raft_consensus.cc:2241] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.551862 340 raft_consensus.cc:2270] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.554081 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3c850021e914530bb106ab7166be7c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:25.554363 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 726d80e05a344f27b5a289723cb5aed7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 726d80e05a344f27b5a289723cb5aed7 already in progress: creating tablet
W20250623 08:04:25.554919 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 already in progress: creating tablet
I20250623 08:04:25.556088 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_3cf27fe1edb94ead83092ffda8c5f7f1 [id=6cc9b03b74284f159ac54dd2dc10dc4b]) successfully deleted
W20250623 08:04:25.557793 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 already in progress: creating tablet
W20250623 08:04:25.558243 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 726d80e05a344f27b5a289723cb5aed7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 726d80e05a344f27b5a289723cb5aed7 already in progress: creating tablet
W20250623 08:04:25.558975 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b already in progress: creating tablet
I20250623 08:04:25.568817 340 ts_tablet_manager.cc:1905] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.569598 344 ts_tablet_manager.cc:1918] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.569732 344 log.cc:1199] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/125784661a7b4e4ba6a710b950e004f8
I20250623 08:04:25.570091 344 ts_tablet_manager.cc:1939] T 125784661a7b4e4ba6a710b950e004f8 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:25.572374 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 125784661a7b4e4ba6a710b950e004f8 (table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]) successfully deleted
I20250623 08:04:25.577364 339 tablet_bootstrap.cc:492] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.577533 339 ts_tablet_manager.cc:1397] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.462s user 0.041s sys 0.095s
W20250623 08:04:25.578045 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet d67a8f1f570741ebb886c4d136ca40d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d67a8f1f570741ebb886c4d136ca40d2 already in progress: creating tablet
W20250623 08:04:25.578362 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 8d8f38a6de1e45e7bed4abdced684c92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d8f38a6de1e45e7bed4abdced684c92 already in progress: creating tablet
W20250623 08:04:25.578634 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 5ca4957928fc4630aa8d8ca6110d7598 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ca4957928fc4630aa8d8ca6110d7598 already in progress: creating tablet
I20250623 08:04:25.468703 32620 tablet_service.cc:1468] Processing CreateTablet for tablet d4221d7500d1458db75926d7b274a654 (DEFAULT_TABLE table=default.Table_110816be0fae488f9cc26b302495009c [id=6de8bb3add89407bb0d3e5685ed5b6bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.470538 343 tablet_bootstrap.cc:654] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.579226 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4221d7500d1458db75926d7b274a654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.495304 341 ts_tablet_manager.cc:1918] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:25.579921 341 log.cc:1199] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/abf018986c24473688c8e5eec724f8ee
I20250623 08:04:25.580152 339 raft_consensus.cc:357] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.580338 339 raft_consensus.cc:383] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.580394 341 ts_tablet_manager.cc:1939] T abf018986c24473688c8e5eec724f8ee P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:25.580401 339 raft_consensus.cc:738] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.580633 339 consensus_queue.cc:260] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.581183 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 85) took 1010 ms. Trace:
I20250623 08:04:25.581332 341 rpcz_store.cc:276] 0623 08:04:24.570537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:24.585530 (+ 14993us) service_pool.cc:225] Handling call
0623 08:04:25.385199 (+799669us) ts_tablet_manager.cc:1145] Deleting tablet abf018986c24473688c8e5eec724f8ee
0623 08:04:25.385206 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:25.392899 (+ 7693us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:25.495161 (+102262us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:25.581136 (+ 85975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":799457}
I20250623 08:04:25.581801 341 tablet_replica.cc:331] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:25.581987 341 raft_consensus.cc:2241] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.582162 341 raft_consensus.cc:2270] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.507267 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 279624b8bb834b95a4f08045968ebfb4 (DEFAULT_TABLE table=default.Table_d000f5be42824b9fad2bdeb33231b5f8 [id=d68afa3a431d4386b862bd4b47a5e1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.509658 32619 tablet_service.cc:1468] Processing CreateTablet for tablet cde7df7d39a54114a4dcb28cfd1eb429 (DEFAULT_TABLE table=default.Table_5174877154cc448c98758389a337e263 [id=e25e0edebbac486785025ef14bcf5352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.583218 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet abf018986c24473688c8e5eec724f8ee (table default.Table_1ef4fa3350714732a2ecec291c255ed6 [id=affccd38bcd54f73af82d0ceeba931f8]) successfully deleted
I20250623 08:04:25.582736 341 ts_tablet_manager.cc:1905] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.583608 339 ts_tablet_manager.cc:1428] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:25.583038 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 279624b8bb834b95a4f08045968ebfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.583813 453 raft_consensus.cc:491] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:25.583926 339 tablet_bootstrap.cc:492] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.583953 453 raft_consensus.cc:513] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.584625 453 leader_election.cc:290] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:25.585302 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7300f526d91f441bb164fe47f3a4f0ce" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:25.585343 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7300f526d91f441bb164fe47f3a4f0ce" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:25.585532 32379 raft_consensus.cc:2466] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:25.586045 32453 leader_election.cc:343] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.586273 32453 leader_election.cc:304] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:25.586634 453 raft_consensus.cc:2802] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:25.586743 453 raft_consensus.cc:491] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:25.586872 339 tablet_bootstrap.cc:654] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.586860 453 raft_consensus.cc:3058] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:25.587227 32619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde7df7d39a54114a4dcb28cfd1eb429. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.587849 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417a050eada4403ebbddff255fb6fb5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.590902 453 raft_consensus.cc:513] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.591982 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7300f526d91f441bb164fe47f3a4f0ce" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:25.592125 453 leader_election.cc:290] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:25.592226 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7300f526d91f441bb164fe47f3a4f0ce" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:25.592408 32379 raft_consensus.cc:3058] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:25.594497 339 tablet_bootstrap.cc:492] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.594672 339 ts_tablet_manager.cc:1397] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
W20250623 08:04:25.595027 32453 leader_election.cc:343] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.595600 339 raft_consensus.cc:357] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.595734 32379 raft_consensus.cc:2466] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:25.596307 32453 leader_election.cc:304] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:25.596832 453 raft_consensus.cc:2802] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:25.595814 339 raft_consensus.cc:383] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.597836 339 raft_consensus.cc:738] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.598419 339 consensus_queue.cc:260] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.534945 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e47b10f46b8402e94354ebca573c194 with delete_type TABLET_DATA_DELETED (replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.535837 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.599609 339 ts_tablet_manager.cc:1428] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:25.599961 339 tablet_bootstrap.cc:492] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.602177 339 tablet_bootstrap.cc:654] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.536036 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ad6cd629c4423ca997efb517cc975b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.607650 341 ts_tablet_manager.cc:1918] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.607800 341 log.cc:1199] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/d3a3c5a446184329b55fcc2cbf969831
I20250623 08:04:25.608206 341 ts_tablet_manager.cc:1939] T d3a3c5a446184329b55fcc2cbf969831 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:25.608711 339 tablet_bootstrap.cc:492] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.608907 339 ts_tablet_manager.cc:1397] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20250623 08:04:25.608918 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 84) took 1038 ms. Trace:
I20250623 08:04:25.609018 341 rpcz_store.cc:276] 0623 08:04:24.570424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:24.588260 (+ 17836us) service_pool.cc:225] Handling call
0623 08:04:25.581742 (+993482us) ts_tablet_manager.cc:1145] Deleting tablet d3a3c5a446184329b55fcc2cbf969831
0623 08:04:25.581750 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:25.597580 (+ 15830us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:25.607556 (+ 9976us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:25.608857 (+ 1301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":993281}
I20250623 08:04:25.609500 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet d3a3c5a446184329b55fcc2cbf969831 (table default.Table_aeb7585a9aad4577b59b1b1164aa016e [id=cf9d65382d8b4fd98f18c628c2ccad23]) successfully deleted
I20250623 08:04:25.609841 339 raft_consensus.cc:357] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.610080 339 raft_consensus.cc:383] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.610167 339 raft_consensus.cc:738] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.611546 344 tablet_replica.cc:331] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.611783 344 raft_consensus.cc:2241] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.612502 339 consensus_queue.cc:260] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.612646 344 raft_consensus.cc:2270] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.613544 453 raft_consensus.cc:695] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:25.613631 339 ts_tablet_manager.cc:1428] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250623 08:04:25.613930 339 tablet_bootstrap.cc:492] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.614401 453 consensus_queue.cc:237] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.615670 339 tablet_bootstrap.cc:654] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:25.621300 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 04a3ae58d26142d58c4df4f52d7e3e2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04a3ae58d26142d58c4df4f52d7e3e2c already in progress: creating tablet
W20250623 08:04:25.627436 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 04eda92e14144733ab71a577dd3bd2b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04eda92e14144733ab71a577dd3bd2b0 already in progress: creating tablet
W20250623 08:04:25.628443 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9918b15e48144ef0b8f3aedc2135be59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9918b15e48144ef0b8f3aedc2135be59 already in progress: creating tablet
W20250623 08:04:25.629442 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 1e0c11ec768a43a48c6dc0f3dd974b71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e0c11ec768a43a48c6dc0f3dd974b71 already in progress: creating tablet
W20250623 08:04:25.630012 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 228375592aee48a1aa21e1d16962b663 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 228375592aee48a1aa21e1d16962b663 already in progress: creating tablet
I20250623 08:04:25.630308 341 tablet_replica.cc:331] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:25.630538 341 raft_consensus.cc:2241] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:25.630823 341 pending_rounds.cc:70] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:25.630890 341 pending_rounds.cc:77] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727383922806784 op_type: NO_OP noop_request { }
I20250623 08:04:25.631016 341 raft_consensus.cc:2887] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:25.631120 341 raft_consensus.cc:2270] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.632341 340 ts_tablet_manager.cc:1918] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250623 08:04:25.632551 343 tablet_bootstrap.cc:492] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:25.632691 343 ts_tablet_manager.cc:1397] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.236s user 0.003s sys 0.008s
I20250623 08:04:25.633452 343 raft_consensus.cc:357] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.633627 343 raft_consensus.cc:383] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.633723 343 raft_consensus.cc:738] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.633946 343 consensus_queue.cc:260] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.635278 344 ts_tablet_manager.cc:1905] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.635445 343 ts_tablet_manager.cc:1428] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:25.637529 343 tablet_bootstrap.cc:492] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:25.639156 343 tablet_bootstrap.cc:654] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.642464 342 tablet_bootstrap.cc:492] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:25.642637 342 ts_tablet_manager.cc:1397] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.144s user 0.011s sys 0.019s
I20250623 08:04:25.645478 32126 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet da430649619c4b26b533dd40a876c63f
I20250623 08:04:25.645553 341 ts_tablet_manager.cc:1905] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.646453 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet c6a97b34e386443fa19a9601621b5017
I20250623 08:04:25.648720 32128 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_284640472f6e45fba6c02c7e87c1ea05" } new_table_name: "default.Table_92e23e1081544f8c9b99f14654981494" modify_external_catalogs: true
I20250623 08:04:25.648656 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_ff8fd153c41246b482ea6b6e3f24c02f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.649358 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff8fd153c41246b482ea6b6e3f24c02f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.650568 32123 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d1cfca29bfb948dbb04297c66d9d2640" } new_table_name: "default.Table_4d4535cab44545839eb3a559385e395a" modify_external_catalogs: true
I20250623 08:04:25.650727 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet d0ee5a4fe03848229524cc0105226d8c
I20250623 08:04:25.651199 32122 catalog_manager.cc:6652] ReplaceTablet: tablet d0ee5a4fe03848229524cc0105226d8c deleted and replaced by tablet 38ecd4c243b840f4b66d8e01078dca64
I20250623 08:04:25.651492 32122 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_11477a913b4b4b20ba2cbdd49dab0cca" } new_table_name: "default.Table_c1bef462c12a418aa2d691eca2e06677" modify_external_catalogs: true
I20250623 08:04:25.651978 32130 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet f0117cbb09dd4067836266c9d6a32adf
I20250623 08:04:25.652523 32130 catalog_manager.cc:6652] ReplaceTablet: tablet f0117cbb09dd4067836266c9d6a32adf deleted and replaced by tablet 2ecf4af9e476495683bd63cac20d4672
I20250623 08:04:25.652849 32130 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_6acaeaf59d9540dcab214d93b801a48e" } new_table_name: "default.Table_509603863ff145f19f99bc7d6743f562" modify_external_catalogs: true
I20250623 08:04:25.654605 32619 tablet_service.cc:1515] Processing DeleteTablet for tablet e6eb8420d9c849a5a751ed38fdac53a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fdc9e53da264a5ca1f70e1846c906f0) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.654856 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet f0117cbb09dd4067836266c9d6a32adf with delete_type TABLET_DATA_DELETED (replaced by tablet 2ecf4af9e476495683bd63cac20d4672) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.655086 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet c6a97b34e386443fa19a9601621b5017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.655354 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet d0ee5a4fe03848229524cc0105226d8c with delete_type TABLET_DATA_DELETED (replaced by tablet 38ecd4c243b840f4b66d8e01078dca64) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.655586 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet f0117cbb09dd4067836266c9d6a32adf with delete_type TABLET_DATA_DELETED (replaced by tablet 2ecf4af9e476495683bd63cac20d4672) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.655788 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet d0ee5a4fe03848229524cc0105226d8c with delete_type TABLET_DATA_DELETED (replaced by tablet 38ecd4c243b840f4b66d8e01078dca64) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.663684 343 tablet_bootstrap.cc:492] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:25.663868 343 ts_tablet_manager.cc:1397] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250623 08:04:25.664717 343 raft_consensus.cc:357] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.664950 343 raft_consensus.cc:383] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.665040 343 raft_consensus.cc:738] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.665287 343 consensus_queue.cc:260] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.665890 343 ts_tablet_manager.cc:1428] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:25.669117 343 tablet_bootstrap.cc:492] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:25.670266 32127 master_service.cc:948] ReplaceTablet: received request to replace tablet b6e29ab07df249748092a2f50a8c0273 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.671319 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet 2795cb14e6db4116a4c9ee539bd4e82d from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.671876 32129 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_2354480bbdc6474bb658478d129314d0" } new_table_name: "default.Table_7b31ef6cd54f4eec82e08a3e592b1dcf" modify_external_catalogs: true
I20250623 08:04:25.672438 341 ts_tablet_manager.cc:1918] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:25.672571 341 log.cc:1199] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/9b6f3fe1b10143aa9b54022546812498
I20250623 08:04:25.673005 341 ts_tablet_manager.cc:1939] T 9b6f3fe1b10143aa9b54022546812498 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:25.673987 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 9b6f3fe1b10143aa9b54022546812498 (table default.Table_d2bb692b2c70474ea4b997cde4fe161a [id=33784d2b4a194d38b56b4881313dc03e]) successfully deleted
I20250623 08:04:25.680547 343 tablet_bootstrap.cc:654] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.681306 32127 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet b6e29ab07df249748092a2f50a8c0273
I20250623 08:04:25.681751 32127 catalog_manager.cc:6652] ReplaceTablet: tablet b6e29ab07df249748092a2f50a8c0273 deleted and replaced by tablet 4cb66c7a273a412db3ae77af9e30d036
I20250623 08:04:25.683046 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet b6e29ab07df249748092a2f50a8c0273 with delete_type TABLET_DATA_DELETED (replaced by tablet 4cb66c7a273a412db3ae77af9e30d036) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.683342 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet b6e29ab07df249748092a2f50a8c0273 with delete_type TABLET_DATA_DELETED (replaced by tablet 4cb66c7a273a412db3ae77af9e30d036) from {username='slave'} at 127.0.0.1:55518
W20250623 08:04:25.684244 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet e90151394fb842498f4e780acab1f1aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e90151394fb842498f4e780acab1f1aa already in progress: creating tablet
W20250623 08:04:25.684733 32128 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35ab9f77a81a4490abb0d2c0c69cd90f consensus state has no leader
W20250623 08:04:25.684867 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 already in progress: creating tablet
W20250623 08:04:25.685406 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 726d80e05a344f27b5a289723cb5aed7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 726d80e05a344f27b5a289723cb5aed7 already in progress: creating tablet
I20250623 08:04:25.687063 341 tablet_replica.cc:331] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:25.687258 341 raft_consensus.cc:2241] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.688563 341 raft_consensus.cc:2270] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.690284 341 ts_tablet_manager.cc:1905] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.691322 539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.69.60:39691 every 8 connection(s)
I20250623 08:04:25.691648 324 rpc_server.cc:307] RPC server started. Bound to: 127.31.69.60:39691
I20250623 08:04:25.688057 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet f0117cbb09dd4067836266c9d6a32adf with delete_type TABLET_DATA_DELETED (replaced by tablet 2ecf4af9e476495683bd63cac20d4672) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.688441 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet b6e29ab07df249748092a2f50a8c0273 with delete_type TABLET_DATA_DELETED (replaced by tablet 4cb66c7a273a412db3ae77af9e30d036) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.688623 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet c6a97b34e386443fa19a9601621b5017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.689184 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet d0ee5a4fe03848229524cc0105226d8c with delete_type TABLET_DATA_DELETED (replaced by tablet 38ecd4c243b840f4b66d8e01078dca64) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.690266 339 tablet_bootstrap.cc:492] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.694036 339 ts_tablet_manager.cc:1397] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.032s
I20250623 08:04:25.694134 340 log.cc:1199] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/99ea7850738743deb7afa87ea2e40daa
I20250623 08:04:25.694702 340 ts_tablet_manager.cc:1939] T 99ea7850738743deb7afa87ea2e40daa P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:25.694074 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet c6a97b34e386443fa19a9601621b5017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:25.695086 32123 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5629d11c44e241d598f3e519b2fe1a44 consensus state has no leader
I20250623 08:04:25.695318 324 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-2/data/info.pb
I20250623 08:04:25.694954 339 raft_consensus.cc:357] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:25.696034 32129 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ac9f38877b24d619c280a1aaa0b7b08 consensus state has no leader
I20250623 08:04:25.696063 339 raft_consensus.cc:383] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.696156 339 raft_consensus.cc:738] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:25.696484 32122 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 175ee90133994154b7e4b58b0770db3b consensus state has no leader
I20250623 08:04:25.696399 339 consensus_queue.cc:260] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:25.696810 32130 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a25bdc9b7e48467fba3771365e58bcbc has no consensus state
I20250623 08:04:25.696941 339 ts_tablet_manager.cc:1428] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:25.697219 339 tablet_bootstrap.cc:492] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.698261 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet ed5c81d9e56c4ee28546cedb9a579f31 with delete_type TABLET_DATA_DELETED (replaced by tablet 04a3ae58d26142d58c4df4f52d7e3e2c) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.698495 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet e90151394fb842498f4e780acab1f1aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.698709 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 125784661a7b4e4ba6a710b950e004f8 with delete_type TABLET_DATA_DELETED (replaced by tablet ae7c2ac546114a819a543f1667fb7844) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.699371 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e0c11ec768a43a48c6dc0f3dd974b71 with delete_type TABLET_DATA_DELETED (replaced by tablet 5629d11c44e241d598f3e519b2fe1a44) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.700034 339 tablet_bootstrap.cc:654] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.701777 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 35428260e05e447ebf88ea41e63cdd02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.701813 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_3cf27fe1edb94ead83092ffda8c5f7f1 [id=6cc9b03b74284f159ac54dd2dc10dc4b]) successfully deleted
I20250623 08:04:25.704687 32020 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskJxinPB/build/asan/bin/kudu as pid 324
I20250623 08:04:25.705036 341 ts_tablet_manager.cc:1918] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250623 08:04:25.705152 341 log.cc:1199] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/99ea7850738743deb7afa87ea2e40daa
I20250623 08:04:25.705605 341 ts_tablet_manager.cc:1939] T 99ea7850738743deb7afa87ea2e40daa P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:25.643674 342 raft_consensus.cc:357] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.710554 342 raft_consensus.cc:383] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.710680 342 raft_consensus.cc:738] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.712576 341 tablet_replica.cc:331] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:25.712548 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 99ea7850738743deb7afa87ea2e40daa (table default.Table_3cf27fe1edb94ead83092ffda8c5f7f1 [id=6cc9b03b74284f159ac54dd2dc10dc4b]) successfully deleted
W20250623 08:04:25.712846 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d already in progress: creating tablet
I20250623 08:04:25.712878 341 raft_consensus.cc:2241] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.713080 341 raft_consensus.cc:2270] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:25.713099 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b already in progress: creating tablet
I20250623 08:04:25.713799 341 ts_tablet_manager.cc:1905] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.714814 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 19536c16ff32460e9db571e4ce411bab (DEFAULT_TABLE table=default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030 [id=f8961c82f9384b0db128a791e00351f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.715237 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19536c16ff32460e9db571e4ce411bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.715977 343 tablet_bootstrap.cc:492] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:25.716157 343 ts_tablet_manager.cc:1397] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.006s
I20250623 08:04:25.717537 343 raft_consensus.cc:357] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.717779 343 raft_consensus.cc:383] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.717893 343 raft_consensus.cc:738] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.718194 340 tablet_replica.cc:331] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:25.718194 343 consensus_queue.cc:260] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.718858 343 ts_tablet_manager.cc:1428] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250623 08:04:25.719161 343 tablet_bootstrap.cc:492] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:25.721117 340 raft_consensus.cc:2241] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:25.721321 343 tablet_bootstrap.cc:654] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.721789 340 pending_rounds.cc:70] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:25.725636 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_e69f9fec5b604bca8658b2124145f92e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:25.725785 342 consensus_queue.cc:260] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.726105 340 pending_rounds.cc:77] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727383232827392 op_type: NO_OP noop_request { }
W20250623 08:04:25.726246 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e69f9fec5b604bca8658b2124145f92e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.726307 340 raft_consensus.cc:2887] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:25.726435 342 ts_tablet_manager.cc:1428] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.084s user 0.003s sys 0.000s
I20250623 08:04:25.728835 342 tablet_bootstrap.cc:492] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:25.728979 340 raft_consensus.cc:2270] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.729904 340 ts_tablet_manager.cc:1905] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.722653 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_baf5c1cd7eee47528cfdd1f4bf5f5d60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.730583 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_baf5c1cd7eee47528cfdd1f4bf5f5d60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.730798 342 tablet_bootstrap.cc:654] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.733346 32130 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_fb91c1ae95e04cefba3aa6b9c00ecc12" } modify_external_catalogs: true
I20250623 08:04:25.733528 32130 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_fb91c1ae95e04cefba3aa6b9c00ecc12" } modify_external_catalogs: true
I20250623 08:04:25.734730 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_8ef30265c1894796843272f538aaa79e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.735270 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ef30265c1894796843272f538aaa79e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.737185 32129 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_e6b4dee87da24acdbfd7d121769b4d86" } modify_external_catalogs: true
I20250623 08:04:25.737427 32129 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_e6b4dee87da24acdbfd7d121769b4d86" } modify_external_catalogs: true
I20250623 08:04:25.737558 32122 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_815c7abc913347a688cc1772704c6715" } modify_external_catalogs: true
I20250623 08:04:25.737736 32122 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_815c7abc913347a688cc1772704c6715" } modify_external_catalogs: true
I20250623 08:04:25.738307 343 tablet_bootstrap.cc:492] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:25.738487 343 ts_tablet_manager.cc:1397] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20250623 08:04:25.738721 32125 master_service.cc:948] ReplaceTablet: received request to replace tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.739308 343 raft_consensus.cc:357] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.739575 343 raft_consensus.cc:383] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.739652 343 raft_consensus.cc:738] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.739919 343 consensus_queue.cc:260] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.740816 343 ts_tablet_manager.cc:1428] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:25.742574 343 tablet_bootstrap.cc:492] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:25.745093 344 ts_tablet_manager.cc:1918] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.747699 343 tablet_bootstrap.cc:654] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.747952 344 log.cc:1199] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/c7385d86316f4a3ab68a77d18080ec20
I20250623 08:04:25.748495 344 ts_tablet_manager.cc:1939] T c7385d86316f4a3ab68a77d18080ec20 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:25.751097 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 19536c16ff32460e9db571e4ce411bab (DEFAULT_TABLE table=default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030 [id=f8961c82f9384b0db128a791e00351f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.751521 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19536c16ff32460e9db571e4ce411bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.752100 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 19536c16ff32460e9db571e4ce411bab (DEFAULT_TABLE table=default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030 [id=f8961c82f9384b0db128a791e00351f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.752460 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19536c16ff32460e9db571e4ce411bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.756294 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet ed5c81d9e56c4ee28546cedb9a579f31 with delete_type TABLET_DATA_DELETED (replaced by tablet 04a3ae58d26142d58c4df4f52d7e3e2c) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.756850 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet c7385d86316f4a3ab68a77d18080ec20 (table default.Table_465c441ac95145f1b498665ccedafe6c [id=24b0c7dcb97e4105b8edf674a0a90e84]) successfully deleted
I20250623 08:04:25.758097 32123 catalog_manager.cc:5582] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:25.758556 340 ts_tablet_manager.cc:1918] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:25.758709 340 log.cc:1199] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/e0c97148429046dc88213ff70a605c99
I20250623 08:04:25.758875 32123 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_b7f7dd030adc46329a94b5de210eb2f9 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 2 got 0
I20250623 08:04:25.759140 340 ts_tablet_manager.cc:1939] T e0c97148429046dc88213ff70a605c99 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:25.759825 32635 tablet_service.cc:1515] Processing DeleteTablet for tablet 125784661a7b4e4ba6a710b950e004f8 with delete_type TABLET_DATA_DELETED (replaced by tablet ae7c2ac546114a819a543f1667fb7844) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.760528 339 tablet_bootstrap.cc:492] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
W20250623 08:04:25.760511 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b already in progress: creating tablet
I20250623 08:04:25.760722 339 ts_tablet_manager.cc:1397] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.003s
W20250623 08:04:25.760879 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d already in progress: creating tablet
I20250623 08:04:25.761112 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet e0c97148429046dc88213ff70a605c99 (table default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]) successfully deleted
I20250623 08:04:25.761750 344 tablet_replica.cc:331] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.761644 339 raft_consensus.cc:357] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.761878 339 raft_consensus.cc:383] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.761960 339 raft_consensus.cc:738] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.761979 344 raft_consensus.cc:2241] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.762161 344 raft_consensus.cc:2270] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.762300 339 consensus_queue.cc:260] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.763442 339 ts_tablet_manager.cc:1428] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:25.764245 344 ts_tablet_manager.cc:1905] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.771241 341 ts_tablet_manager.cc:1918] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.771451 341 log.cc:1199] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/e0c97148429046dc88213ff70a605c99
I20250623 08:04:25.771872 341 ts_tablet_manager.cc:1939] T e0c97148429046dc88213ff70a605c99 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:25.772626 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 7419cb91360f49e1b3cb7e90732558c5 (DEFAULT_TABLE table=default.Table_1da3c8ca68084386b7f0d95a6f85927a [id=7cc0919082d04fc4a02aa856a42e5529]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.773118 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7419cb91360f49e1b3cb7e90732558c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.776185 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet e0c97148429046dc88213ff70a605c99 (table default.Table_52580ab1940b40b0b31f4b95f9993f67 [id=2fd444ee4cf340118edab36597dc75f8]) successfully deleted
I20250623 08:04:25.777294 341 tablet_replica.cc:331] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:25.777503 341 raft_consensus.cc:2241] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.777698 341 raft_consensus.cc:2270] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.778133 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 7419cb91360f49e1b3cb7e90732558c5 (DEFAULT_TABLE table=default.Table_1da3c8ca68084386b7f0d95a6f85927a [id=7cc0919082d04fc4a02aa856a42e5529]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.778591 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7419cb91360f49e1b3cb7e90732558c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.782393 362 raft_consensus.cc:491] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:25.782572 362 raft_consensus.cc:513] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.782869 341 ts_tablet_manager.cc:1905] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.783366 362 leader_election.cc:290] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:25.784125 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04a3ae58d26142d58c4df4f52d7e3e2c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:25.784202 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 7419cb91360f49e1b3cb7e90732558c5 (DEFAULT_TABLE table=default.Table_1da3c8ca68084386b7f0d95a6f85927a [id=7cc0919082d04fc4a02aa856a42e5529]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.784569 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04a3ae58d26142d58c4df4f52d7e3e2c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:25.784647 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 74842fde5152419ca35f3950e465247a (DEFAULT_TABLE table=default.Table_af3136dbadfb43c892a197cc3a5d684e [id=2bfacaaab9f347b1a77e5eb29a9c485e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.784686 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 32c82141edea4068b54755a85c6d7fb8 (DEFAULT_TABLE table=default.Table_9788bd7b3d2848368b935832a2cffaed [id=7e59ad2235c1418d97f2b73ff2a48300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.784999 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74842fde5152419ca35f3950e465247a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.785091 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c82141edea4068b54755a85c6d7fb8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:25.785320 32314 leader_election.cc:343] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.789647 32130 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f1c62741c7ec4aa482c4970168e28c01
I20250623 08:04:25.789769 32129 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 136ca351c0944519843756152e999ee8
I20250623 08:04:25.790127 32125 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa
I20250623 08:04:25.790648 32125 catalog_manager.cc:6652] ReplaceTablet: tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa deleted and replaced by tablet 6f9e0d1b85a94ae9ae86e53688a55db5
I20250623 08:04:25.791410 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 32c82141edea4068b54755a85c6d7fb8 (DEFAULT_TABLE table=default.Table_9788bd7b3d2848368b935832a2cffaed [id=7e59ad2235c1418d97f2b73ff2a48300]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:25.792109 32314 leader_election.cc:343] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.792271 32314 leader_election.cc:304] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:25.792711 362 raft_consensus.cc:2747] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:25.793676 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f9e0d1b85a94ae9ae86e53688a55db5) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.794066 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 67c44697ecab4dd9b43f544a602897ca
I20250623 08:04:25.794266 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f9e0d1b85a94ae9ae86e53688a55db5) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.794682 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f9e0d1b85a94ae9ae86e53688a55db5) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.797909 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet 67c44697ecab4dd9b43f544a602897ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.798542 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 32c82141edea4068b54755a85c6d7fb8 (DEFAULT_TABLE table=default.Table_9788bd7b3d2848368b935832a2cffaed [id=7e59ad2235c1418d97f2b73ff2a48300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.799041 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c82141edea4068b54755a85c6d7fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.800412 339 tablet_bootstrap.cc:492] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.801985 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7419cb91360f49e1b3cb7e90732558c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.803194 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet 67c44697ecab4dd9b43f544a602897ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:25.803767 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 67c44697ecab4dd9b43f544a602897ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:25.805070 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 74842fde5152419ca35f3950e465247a (DEFAULT_TABLE table=default.Table_af3136dbadfb43c892a197cc3a5d684e [id=2bfacaaab9f347b1a77e5eb29a9c485e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.805469 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74842fde5152419ca35f3950e465247a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.808305 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c82141edea4068b54755a85c6d7fb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.814721 32499 tablet_service.cc:1468] Processing CreateTablet for tablet a25bdc9b7e48467fba3771365e58bcbc (DEFAULT_TABLE table=default.Table_509603863ff145f19f99bc7d6743f562 [id=149914e63b704358866228fb74895591]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.815030 32426 heartbeater.cc:344] Connected to a master server at 127.31.69.60:39691
I20250623 08:04:25.815164 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a25bdc9b7e48467fba3771365e58bcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.816074 32351 tablet_service.cc:1468] Processing CreateTablet for tablet ba3d2fa01d574dd992dc904ac024ba3a (DEFAULT_TABLE table=default.Table_4acf3aff222e477a885bc37d00623ddf [id=6c57db68f5ed45a3b64d1e3a02ce8a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.816376 362 raft_consensus.cc:491] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:25.816468 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba3d2fa01d574dd992dc904ac024ba3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.816556 362 raft_consensus.cc:513] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.817281 362 leader_election.cc:290] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:25.818368 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9d871507dcc46fc80f6e0ac739c6476" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:25.818943 32314 leader_election.cc:343] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: b9d871507dcc46fc80f6e0ac739c6476
I20250623 08:04:25.819047 32354 tablet_service.cc:1468] Processing CreateTablet for tablet a25bdc9b7e48467fba3771365e58bcbc (DEFAULT_TABLE table=default.Table_509603863ff145f19f99bc7d6743f562 [id=149914e63b704358866228fb74895591]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:25.819214 32314 leader_election.cc:343] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.819314 32314 leader_election.cc:304] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:25.819437 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a25bdc9b7e48467fba3771365e58bcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.819715 362 raft_consensus.cc:2747] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:25.819968 340 tablet_replica.cc:331] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:25.820159 340 raft_consensus.cc:2241] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.820384 340 raft_consensus.cc:2270] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.820775 344 ts_tablet_manager.cc:1918] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.820897 344 log.cc:1199] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a46c0a89fc014054a75ed198e824b5b6
I20250623 08:04:25.820993 340 ts_tablet_manager.cc:1905] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.821256 344 ts_tablet_manager.cc:1939] T a46c0a89fc014054a75ed198e824b5b6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:25.821540 32630 tablet_service.cc:1468] Processing CreateTablet for tablet ba3d2fa01d574dd992dc904ac024ba3a (DEFAULT_TABLE table=default.Table_4acf3aff222e477a885bc37d00623ddf [id=6c57db68f5ed45a3b64d1e3a02ce8a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.821969 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba3d2fa01d574dd992dc904ac024ba3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.822655 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet a46c0a89fc014054a75ed198e824b5b6 (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]) successfully deleted
I20250623 08:04:25.823879 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 67c44697ecab4dd9b43f544a602897ca (DEFAULT_TABLE table=default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.824414 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67c44697ecab4dd9b43f544a602897ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.827051 32498 tablet_service.cc:1468] Processing CreateTablet for tablet ba3d2fa01d574dd992dc904ac024ba3a (DEFAULT_TABLE table=default.Table_4acf3aff222e477a885bc37d00623ddf [id=6c57db68f5ed45a3b64d1e3a02ce8a2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.827468 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba3d2fa01d574dd992dc904ac024ba3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.827898 342 tablet_bootstrap.cc:492] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:25.828068 342 ts_tablet_manager.cc:1397] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.008s
I20250623 08:04:25.828843 342 raft_consensus.cc:357] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.829067 342 raft_consensus.cc:383] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.829154 342 raft_consensus.cc:738] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.829454 342 consensus_queue.cc:260] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.830152 342 ts_tablet_manager.cc:1428] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250623 08:04:25.830440 342 tablet_bootstrap.cc:492] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:25.831449 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 67c44697ecab4dd9b43f544a602897ca (DEFAULT_TABLE table=default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.831784 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67c44697ecab4dd9b43f544a602897ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.832012 342 tablet_bootstrap.cc:654] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.834674 344 tablet_replica.cc:331] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.837033 540 sys_catalog.cc:263] Verifying existing consensus state
I20250623 08:04:25.838096 341 ts_tablet_manager.cc:1918] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.838229 341 log.cc:1199] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b1db513f73d24116af299da4d267aabf
I20250623 08:04:25.838613 341 ts_tablet_manager.cc:1939] T b1db513f73d24116af299da4d267aabf P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:25.840845 344 raft_consensus.cc:2241] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.841037 344 raft_consensus.cc:2270] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.841619 540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: Bootstrap starting.
I20250623 08:04:25.842028 344 ts_tablet_manager.cc:1905] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.843933 32566 heartbeater.cc:344] Connected to a master server at 127.31.69.60:39691
I20250623 08:04:25.848397 32701 heartbeater.cc:344] Connected to a master server at 127.31.69.60:39691
I20250623 08:04:25.848623 32701 heartbeater.cc:499] Master 127.31.69.60:39691 was elected leader, sending a full tablet report...
I20250623 08:04:25.849231 341 tablet_replica.cc:331] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:25.849469 341 raft_consensus.cc:2241] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.849643 341 raft_consensus.cc:2270] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.850519 341 ts_tablet_manager.cc:1905] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.821540 32636 tablet_service.cc:1468] Processing CreateTablet for tablet a25bdc9b7e48467fba3771365e58bcbc (DEFAULT_TABLE table=default.Table_509603863ff145f19f99bc7d6743f562 [id=149914e63b704358866228fb74895591]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.852144 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a25bdc9b7e48467fba3771365e58bcbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.855618 32701 heartbeater.cc:499] Master 127.31.69.60:39691 was elected leader, sending a full tablet report...
I20250623 08:04:25.856113 540 log.cc:826] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: Log is configured to *not* fsync() on all Append() calls
I20250623 08:04:25.857504 341 ts_tablet_manager.cc:1918] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.857687 341 log.cc:1199] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/e6eb8420d9c849a5a751ed38fdac53a4
I20250623 08:04:25.859810 339 tablet_bootstrap.cc:654] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.860446 342 tablet_bootstrap.cc:492] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:25.860666 342 ts_tablet_manager.cc:1397] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
W20250623 08:04:25.822947 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 1e47b10f46b8402e94354ebca573c194 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e47b10f46b8402e94354ebca573c194 already in progress: creating tablet
I20250623 08:04:25.861543 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b1db513f73d24116af299da4d267aabf (table default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652]) successfully deleted
W20250623 08:04:25.861959 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d67a8f1f570741ebb886c4d136ca40d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d67a8f1f570741ebb886c4d136ca40d2 already in progress: creating tablet
I20250623 08:04:25.861930 342 raft_consensus.cc:357] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.862128 342 raft_consensus.cc:383] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 08:04:25.862387 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 8d8f38a6de1e45e7bed4abdced684c92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d8f38a6de1e45e7bed4abdced684c92 already in progress: creating tablet
I20250623 08:04:25.862255 342 raft_consensus.cc:738] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
W20250623 08:04:25.862840 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 2795cb14e6db4116a4c9ee539bd4e82d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2795cb14e6db4116a4c9ee539bd4e82d already in progress: creating tablet
W20250623 08:04:25.863517 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 1e47b10f46b8402e94354ebca573c194 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e47b10f46b8402e94354ebca573c194 already in progress: creating tablet
W20250623 08:04:25.863806 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b6ad6cd629c4423ca997efb517cc975b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6ad6cd629c4423ca997efb517cc975b already in progress: creating tablet
I20250623 08:04:25.865322 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet 04a3ae58d26142d58c4df4f52d7e3e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:25.865832 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 74842fde5152419ca35f3950e465247a (DEFAULT_TABLE table=default.Table_af3136dbadfb43c892a197cc3a5d684e [id=2bfacaaab9f347b1a77e5eb29a9c485e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.866274 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74842fde5152419ca35f3950e465247a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.867549 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_a615b39019e1478aa69bd46fc2e4a4a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.868129 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a615b39019e1478aa69bd46fc2e4a4a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.869390 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_a22e12b2bd5b44aab29760196b28bd06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.869876 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a22e12b2bd5b44aab29760196b28bd06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.870994 32124 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_110816be0fae488f9cc26b302495009c" } modify_external_catalogs: true
I20250623 08:04:25.871169 32124 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_110816be0fae488f9cc26b302495009c" } modify_external_catalogs: true
I20250623 08:04:25.872361 340 ts_tablet_manager.cc:1918] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.872470 340 log.cc:1199] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b9d871507dcc46fc80f6e0ac739c6476
I20250623 08:04:25.872812 340 ts_tablet_manager.cc:1939] T b9d871507dcc46fc80f6e0ac739c6476 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:25.874979 340 tablet_replica.cc:331] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:25.875169 340 raft_consensus.cc:2241] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:25.875381 340 pending_rounds.cc:70] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:25.875449 340 pending_rounds.cc:77] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727385074126848 op_type: NO_OP noop_request { }
I20250623 08:04:25.875552 340 raft_consensus.cc:2887] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:25.877435 32354 tablet_service.cc:1468] Processing CreateTablet for tablet b81263a9ee9d4da1ac721c3b2bd21248 (DEFAULT_TABLE table=default.Table_4e377b569121400d82096403060f9cda [id=49bfa95de97b443ba51225f84820fcf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.877797 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b81263a9ee9d4da1ac721c3b2bd21248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.878964 32126 master_service.cc:948] ReplaceTablet: received request to replace tablet 35ab9f77a81a4490abb0d2c0c69cd90f from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:25.879133 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_7f123c9cf6f54d22ae18a9d6ce7d9d9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.879688 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f123c9cf6f54d22ae18a9d6ce7d9d9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.880285 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_c382db2a0c144ec4abd51b3e927e0a17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.880756 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c382db2a0c144ec4abd51b3e927e0a17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.880843 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_48cbdc58fd034e6390087ed7c6346d24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:25.881335 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48cbdc58fd034e6390087ed7c6346d24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.882304 32496 tablet_service.cc:1468] Processing CreateTablet for tablet b81263a9ee9d4da1ac721c3b2bd21248 (DEFAULT_TABLE table=default.Table_4e377b569121400d82096403060f9cda [id=49bfa95de97b443ba51225f84820fcf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.882572 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_4c5e4af6e0da4344a426fc6bcfafdc97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:25.882731 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b81263a9ee9d4da1ac721c3b2bd21248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.882674 342 consensus_queue.cc:260] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:25.883126 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c5e4af6e0da4344a426fc6bcfafdc97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:25.883967 342 ts_tablet_manager.cc:1428] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20250623 08:04:25.875689 340 raft_consensus.cc:2270] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.885039 342 tablet_bootstrap.cc:492] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:25.886504 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_9b48914a8da94066afd2e04c5e639a95" } new_table_name: "default.Table_464d0877823d4ed399bfd7caf198511d" modify_external_catalogs: true
I20250623 08:04:25.887398 32123 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f47e2bf1990949db8996199686e7034f" } modify_external_catalogs: true
I20250623 08:04:25.887547 32123 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f47e2bf1990949db8996199686e7034f" } modify_external_catalogs: true
I20250623 08:04:25.887892 32498 tablet_service.cc:1468] Processing CreateTablet for tablet d02f74c24a174b18b08a870a4e47fefb (DEFAULT_TABLE table=default.Table_8f909d419b67443f82c2c2d381c84a99 [id=b34c2a31f23742838fccd1bb73863d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.886468 32358 tablet_service.cc:1468] Processing CreateTablet for tablet d02f74c24a174b18b08a870a4e47fefb (DEFAULT_TABLE table=default.Table_8f909d419b67443f82c2c2d381c84a99 [id=b34c2a31f23742838fccd1bb73863d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.888681 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d02f74c24a174b18b08a870a4e47fefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.888939 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d02f74c24a174b18b08a870a4e47fefb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:25.890223 32116 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 175ee90133994154b7e4b58b0770db3b (table default.Table_c1bef462c12a418aa2d691eca2e06677 [id=426f83a7100640699646c6a02f20c6cc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 175ee90133994154b7e4b58b0770db3b consensus state has no leader
I20250623 08:04:25.887423 32353 tablet_service.cc:1468] Processing CreateTablet for tablet c7fcf583064f4538913d9157e93068dd (DEFAULT_TABLE table=default.Table_c4287575a32b4847ba2379ef60074ba9 [id=abda89a30971466ea9a1f410d424ef58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.891438 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7fcf583064f4538913d9157e93068dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.891978 342 tablet_bootstrap.cc:654] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.888259 339 tablet_bootstrap.cc:492] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.904417 339 ts_tablet_manager.cc:1397] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.002s
I20250623 08:04:25.905282 339 raft_consensus.cc:357] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.905467 339 raft_consensus.cc:383] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.905567 339 raft_consensus.cc:738] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.905674 344 ts_tablet_manager.cc:1918] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.905838 339 consensus_queue.cc:260] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.905993 344 log.cc:1199] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b6ad6cd629c4423ca997efb517cc975b
I20250623 08:04:25.906795 341 ts_tablet_manager.cc:1939] T e6eb8420d9c849a5a751ed38fdac53a4 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:25.907099 339 ts_tablet_manager.cc:1428] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:25.907395 339 tablet_bootstrap.cc:492] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.909025 339 tablet_bootstrap.cc:654] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.910100 344 ts_tablet_manager.cc:1939] T b6ad6cd629c4423ca997efb517cc975b P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:25.885535 340 ts_tablet_manager.cc:1905] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.910908 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 31a67b70cbc64bb3bfd5a34865109a01 (DEFAULT_TABLE table=default.Table_463abc7f8321496da24ccb06cba92d25 [id=7a569a9fe3bd4c608ddca48a734df1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.911279 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a67b70cbc64bb3bfd5a34865109a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.912151 32499 tablet_service.cc:1468] Processing CreateTablet for tablet c7fcf583064f4538913d9157e93068dd (DEFAULT_TABLE table=default.Table_c4287575a32b4847ba2379ef60074ba9 [id=abda89a30971466ea9a1f410d424ef58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.912340 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 31a67b70cbc64bb3bfd5a34865109a01 (DEFAULT_TABLE table=default.Table_463abc7f8321496da24ccb06cba92d25 [id=7a569a9fe3bd4c608ddca48a734df1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.912562 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7fcf583064f4538913d9157e93068dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.912662 453 raft_consensus.cc:491] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:25.914031 453 raft_consensus.cc:513] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:25.916522 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 2795cb14e6db4116a4c9ee539bd4e82d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2795cb14e6db4116a4c9ee539bd4e82d already in progress: creating tablet
I20250623 08:04:25.917805 343 tablet_bootstrap.cc:492] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:25.917989 343 ts_tablet_manager.cc:1397] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.176s user 0.012s sys 0.002s
I20250623 08:04:25.918082 453 leader_election.cc:290] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:25.919639 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c77fd6e54904722acb2d1437c627ba8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:25.919889 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 432d1642c0e44119b5bc05858463a136 (DEFAULT_TABLE table=default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.920423 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432d1642c0e44119b5bc05858463a136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.920979 344 tablet_replica.cc:331] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.921262 344 raft_consensus.cc:2241] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.921284 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 432d1642c0e44119b5bc05858463a136 (DEFAULT_TABLE table=default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:25.921703 32453 leader_election.cc:343] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.921972 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a67b70cbc64bb3bfd5a34865109a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.922477 344 raft_consensus.cc:2270] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.922986 344 ts_tablet_manager.cc:1905] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.927441 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432d1642c0e44119b5bc05858463a136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.936702 339 tablet_bootstrap.cc:492] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.936890 339 ts_tablet_manager.cc:1397] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250623 08:04:25.936968 540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=24 ignored=0} inserts{seen=23 ignored=0} mutations{seen=14 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250623 08:04:25.937712 540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: Bootstrap complete.
I20250623 08:04:25.937929 339 raft_consensus.cc:357] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.938156 339 raft_consensus.cc:383] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.938262 339 raft_consensus.cc:738] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.938552 339 consensus_queue.cc:260] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.939276 339 ts_tablet_manager.cc:1428] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:25.939654 339 tablet_bootstrap.cc:492] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.941561 339 tablet_bootstrap.cc:654] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.946681 344 ts_tablet_manager.cc:1918] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:25.958667 339 tablet_bootstrap.cc:492] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.958853 339 ts_tablet_manager.cc:1397] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.004s
I20250623 08:04:25.959642 339 raft_consensus.cc:357] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.959834 339 raft_consensus.cc:383] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.959931 339 raft_consensus.cc:738] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.960207 339 consensus_queue.cc:260] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.960815 339 ts_tablet_manager.cc:1428] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:25.961127 339 tablet_bootstrap.cc:492] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.962895 339 tablet_bootstrap.cc:654] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.959540 540 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:25.968266 362 raft_consensus.cc:491] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:25.968428 362 raft_consensus.cc:513] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:25.969028 362 leader_election.cc:290] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:25.970021 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c1596e80225041099fb513ce477eae33" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:25.975534 344 log.cc:1199] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/e90151394fb842498f4e780acab1f1aa
I20250623 08:04:25.975951 344 ts_tablet_manager.cc:1939] T e90151394fb842498f4e780acab1f1aa P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:25.976032 32314 leader_election.cc:343] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.977178 540 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6644af0f4b2a4b998282b021ec1dc678, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.978097 344 tablet_replica.cc:331] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:25.978327 344 raft_consensus.cc:2241] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:25.978315 540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:25.978555 344 raft_consensus.cc:2270] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:25.979137 344 ts_tablet_manager.cc:1905] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:25.979171 339 tablet_bootstrap.cc:492] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:25.979378 339 ts_tablet_manager.cc:1397] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20250623 08:04:25.980202 339 raft_consensus.cc:357] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.980465 339 raft_consensus.cc:383] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.980595 339 raft_consensus.cc:738] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.980981 339 consensus_queue.cc:260] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:25.981645 339 ts_tablet_manager.cc:1428] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:25.981985 339 tablet_bootstrap.cc:492] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:25.983729 339 tablet_bootstrap.cc:654] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:25.987509 366 raft_consensus.cc:491] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:25.987326 32426 heartbeater.cc:344] Connected to a master server at 127.31.69.60:39691
I20250623 08:04:25.987696 366 raft_consensus.cc:513] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.989543 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c1596e80225041099fb513ce477eae33" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:25.989601 32589 leader_election.cc:343] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 1e0c11ec768a43a48c6dc0f3dd974b71
I20250623 08:04:25.989930 32633 tablet_service.cc:1468] Processing CreateTablet for tablet b81263a9ee9d4da1ac721c3b2bd21248 (DEFAULT_TABLE table=default.Table_4e377b569121400d82096403060f9cda [id=49bfa95de97b443ba51225f84820fcf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.990718 366 leader_election.cc:290] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:25.990813 32314 leader_election.cc:343] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.990962 32314 leader_election.cc:304] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:25.991391 362 raft_consensus.cc:2747] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:25.992318 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b81263a9ee9d4da1ac721c3b2bd21248. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:25.992622 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c77fd6e54904722acb2d1437c627ba8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:25.992708 560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:25.993021 560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:25.994357 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 67c44697ecab4dd9b43f544a602897ca (DEFAULT_TABLE table=default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:25.995229 343 raft_consensus.cc:357] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.995288 540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 08:04:25.995391 343 raft_consensus.cc:383] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:25.995464 343 raft_consensus.cc:738] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.995662 343 consensus_queue.cc:260] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:25.996125 343 ts_tablet_manager.cc:1428] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.078s user 0.002s sys 0.000s
W20250623 08:04:25.996620 32453 leader_election.cc:343] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:25.996804 32453 leader_election.cc:304] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:25.997470 453 raft_consensus.cc:2747] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:26.005786 32589 leader_election.cc:343] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 1e0c11ec768a43a48c6dc0f3dd974b71
I20250623 08:04:26.005954 32589 leader_election.cc:304] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:26.006433 366 raft_consensus.cc:2747] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:26.014716 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67c44697ecab4dd9b43f544a602897ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.015993 341 tablet_replica.cc:331] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:26.016239 341 raft_consensus.cc:2241] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.016513 341 raft_consensus.cc:2270] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.020342 341 ts_tablet_manager.cc:1905] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.025784 32635 tablet_service.cc:1468] Processing CreateTablet for tablet d02f74c24a174b18b08a870a4e47fefb (DEFAULT_TABLE table=default.Table_8f909d419b67443f82c2c2d381c84a99 [id=b34c2a31f23742838fccd1bb73863d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.026183 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d02f74c24a174b18b08a870a4e47fefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.026525 32619 tablet_service.cc:1468] Processing CreateTablet for tablet c7fcf583064f4538913d9157e93068dd (DEFAULT_TABLE table=default.Table_c4287575a32b4847ba2379ef60074ba9 [id=abda89a30971466ea9a1f410d424ef58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.026932 32619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7fcf583064f4538913d9157e93068dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.027678 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 31a67b70cbc64bb3bfd5a34865109a01 (DEFAULT_TABLE table=default.Table_463abc7f8321496da24ccb06cba92d25 [id=7a569a9fe3bd4c608ddca48a734df1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.027995 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a67b70cbc64bb3bfd5a34865109a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.028695 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 432d1642c0e44119b5bc05858463a136 (DEFAULT_TABLE table=default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.029103 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432d1642c0e44119b5bc05858463a136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.030550 340 ts_tablet_manager.cc:1918] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:26.030870 343 tablet_bootstrap.cc:492] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.030712 340 log.cc:1199] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/c7385d86316f4a3ab68a77d18080ec20
I20250623 08:04:26.032953 343 tablet_bootstrap.cc:654] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.033118 340 ts_tablet_manager.cc:1939] T c7385d86316f4a3ab68a77d18080ec20 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.036083 340 tablet_replica.cc:331] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.036150 342 tablet_bootstrap.cc:492] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:26.036310 340 raft_consensus.cc:2241] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.036372 342 ts_tablet_manager.cc:1397] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.152s user 0.037s sys 0.053s
I20250623 08:04:26.036475 340 raft_consensus.cc:2270] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.037386 342 raft_consensus.cc:357] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.037685 342 raft_consensus.cc:383] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.037782 342 raft_consensus.cc:738] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.037938 340 ts_tablet_manager.cc:1905] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.038064 342 consensus_queue.cc:260] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.039033 342 ts_tablet_manager.cc:1428] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250623 08:04:26.039481 342 tablet_bootstrap.cc:492] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:26.041785 342 tablet_bootstrap.cc:654] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.046598 340 ts_tablet_manager.cc:1918] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.046769 340 log.cc:1199] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/5ca4957928fc4630aa8d8ca6110d7598
I20250623 08:04:26.049551 342 tablet_bootstrap.cc:492] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:26.049685 342 ts_tablet_manager.cc:1397] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20250623 08:04:26.050662 342 raft_consensus.cc:357] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.050900 342 raft_consensus.cc:383] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.051002 342 raft_consensus.cc:738] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.051324 342 consensus_queue.cc:260] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.051942 342 ts_tablet_manager.cc:1428] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:26.052320 342 tablet_bootstrap.cc:492] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:26.052659 340 ts_tablet_manager.cc:1939] T 5ca4957928fc4630aa8d8ca6110d7598 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.054478 342 tablet_bootstrap.cc:654] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.056337 340 tablet_replica.cc:331] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.056522 340 raft_consensus.cc:2241] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.056690 340 raft_consensus.cc:2270] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.058012 340 ts_tablet_manager.cc:1905] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.059945 342 tablet_bootstrap.cc:492] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:26.060096 342 ts_tablet_manager.cc:1397] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20250623 08:04:26.060804 342 raft_consensus.cc:357] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.060999 342 raft_consensus.cc:383] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.061069 342 raft_consensus.cc:738] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.061280 342 consensus_queue.cc:260] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.061686 342 ts_tablet_manager.cc:1428] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:26.061975 342 tablet_bootstrap.cc:492] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:26.063861 342 tablet_bootstrap.cc:654] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.066107 340 ts_tablet_manager.cc:1918] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.066241 340 log.cc:1199] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b6ad6cd629c4423ca997efb517cc975b
I20250623 08:04:26.066675 340 ts_tablet_manager.cc:1939] T b6ad6cd629c4423ca997efb517cc975b P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.069774 340 tablet_replica.cc:331] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.070050 340 raft_consensus.cc:2241] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:26.070483 340 raft_consensus.cc:2270] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.071377 340 ts_tablet_manager.cc:1905] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.079453 340 ts_tablet_manager.cc:1918] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:26.079620 340 log.cc:1199] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b6e29ab07df249748092a2f50a8c0273
I20250623 08:04:26.080078 340 ts_tablet_manager.cc:1939] T b6e29ab07df249748092a2f50a8c0273 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.081511 340 tablet_replica.cc:331] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.081753 340 raft_consensus.cc:2241] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.081950 340 raft_consensus.cc:2270] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.082893 340 ts_tablet_manager.cc:1905] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.090852 340 ts_tablet_manager.cc:1918] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.091015 340 log.cc:1199] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/ed5c81d9e56c4ee28546cedb9a579f31
I20250623 08:04:26.091459 340 ts_tablet_manager.cc:1939] T ed5c81d9e56c4ee28546cedb9a579f31 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.092007 32701 heartbeater.cc:344] Connected to a master server at 127.31.69.60:39691
I20250623 08:04:26.092234 32701 heartbeater.cc:499] Master 127.31.69.60:39691 was elected leader, sending a full tablet report...
I20250623 08:04:26.097534 340 tablet_replica.cc:331] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.097793 340 raft_consensus.cc:2241] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.097972 340 raft_consensus.cc:2270] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.098835 340 ts_tablet_manager.cc:1905] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.100296 453 raft_consensus.cc:491] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.100530 453 raft_consensus.cc:513] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.101387 453 leader_election.cc:290] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:26.102334 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69e772df590747ff841be5f218e55acf" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:26.102546 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69e772df590747ff841be5f218e55acf" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:26.103225 32453 leader_election.cc:343] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:26.103705 32453 leader_election.cc:343] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:26.103839 32453 leader_election.cc:304] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:26.104532 453 raft_consensus.cc:2747] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:26.108727 340 ts_tablet_manager.cc:1918] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.108896 340 log.cc:1199] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/e90151394fb842498f4e780acab1f1aa
I20250623 08:04:26.109273 340 ts_tablet_manager.cc:1939] T e90151394fb842498f4e780acab1f1aa P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.110445 340 tablet_replica.cc:331] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.110687 340 raft_consensus.cc:2241] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.110944 340 raft_consensus.cc:2270] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.111555 343 tablet_bootstrap.cc:492] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.111567 340 ts_tablet_manager.cc:1905] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.111737 343 ts_tablet_manager.cc:1397] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.000s
I20250623 08:04:26.112550 343 raft_consensus.cc:357] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.112758 343 raft_consensus.cc:383] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.112847 343 raft_consensus.cc:738] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.113142 343 consensus_queue.cc:260] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.113741 32701 heartbeater.cc:499] Master 127.31.69.60:39691 was elected leader, sending a full tablet report...
I20250623 08:04:26.113798 343 ts_tablet_manager.cc:1428] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:26.114253 343 tablet_bootstrap.cc:492] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.116092 343 tablet_bootstrap.cc:654] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.127221 343 tablet_bootstrap.cc:492] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.127388 343 ts_tablet_manager.cc:1397] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20250623 08:04:26.128158 343 raft_consensus.cc:357] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.128366 343 raft_consensus.cc:383] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.128461 343 raft_consensus.cc:738] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:25.921206 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b6ad6cd629c4423ca997efb517cc975b (table default.Table_47264042125847ac997923572c293a3e [id=4214b1f0ca1744dc9b46d3f4d7618ba3]) successfully deleted
W20250623 08:04:26.129703 32114 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 208416us
I20250623 08:04:26.129963 32701 heartbeater.cc:499] Master 127.31.69.60:39691 was elected leader, sending a full tablet report...
I20250623 08:04:26.130455 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet e90151394fb842498f4e780acab1f1aa (table default.Table_eefa8d393087436793f03c22b76ab152 [id=a1d4326f22fa44ba82cb376970f55ddc]) successfully deleted
W20250623 08:04:26.130779 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet c6a97b34e386443fa19a9601621b5017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a97b34e386443fa19a9601621b5017 already in progress: creating tablet
W20250623 08:04:26.131045 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet d0ee5a4fe03848229524cc0105226d8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0ee5a4fe03848229524cc0105226d8c already in progress: creating tablet
W20250623 08:04:26.131297 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet f0117cbb09dd4067836266c9d6a32adf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0117cbb09dd4067836266c9d6a32adf already in progress: creating tablet
I20250623 08:04:26.132032 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c77fd6e54904722acb2d1437c627ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0861107ab4624a05bd7a4d24b420c502) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.136938 341 ts_tablet_manager.cc:1918] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:26.137091 341 log.cc:1199] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b6e29ab07df249748092a2f50a8c0273
I20250623 08:04:26.140359 340 ts_tablet_manager.cc:1918] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.140594 340 log.cc:1199] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/125784661a7b4e4ba6a710b950e004f8
I20250623 08:04:26.141039 340 ts_tablet_manager.cc:1939] T 125784661a7b4e4ba6a710b950e004f8 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.141968 32379 raft_consensus.cc:1273] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Refusing update from remote peer 59f7c614254f42009306b13337f06576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 08:04:26.142830 453 consensus_queue.cc:1035] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:26.142984 340 tablet_replica.cc:331] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.143255 340 raft_consensus.cc:2241] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.143667 340 raft_consensus.cc:2270] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.146857 343 consensus_queue.cc:260] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.148238 343 ts_tablet_manager.cc:1428] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.021s user 0.002s sys 0.001s
I20250623 08:04:26.148550 343 tablet_bootstrap.cc:492] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.150377 343 tablet_bootstrap.cc:654] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.155997 343 tablet_bootstrap.cc:492] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.156179 343 ts_tablet_manager.cc:1397] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250623 08:04:26.157178 343 raft_consensus.cc:357] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.157456 32701 heartbeater.cc:499] Master 127.31.69.60:39691 was elected leader, sending a full tablet report...
I20250623 08:04:26.137579 341 ts_tablet_manager.cc:1939] T b6e29ab07df249748092a2f50a8c0273 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:26.163811 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet e6eb8420d9c849a5a751ed38fdac53a4 (table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]) successfully deleted
I20250623 08:04:26.164022 343 raft_consensus.cc:383] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.164134 343 raft_consensus.cc:738] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:26.164144 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f0117cbb09dd4067836266c9d6a32adf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0117cbb09dd4067836266c9d6a32adf already in progress: creating tablet
W20250623 08:04:26.164355 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d0ee5a4fe03848229524cc0105226d8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0ee5a4fe03848229524cc0105226d8c already in progress: creating tablet
I20250623 08:04:26.165273 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]) successfully deleted
I20250623 08:04:26.164891 343 consensus_queue.cc:260] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:26.165510 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet c6a97b34e386443fa19a9601621b5017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a97b34e386443fa19a9601621b5017 already in progress: creating tablet
I20250623 08:04:26.166183 343 ts_tablet_manager.cc:1428] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250623 08:04:26.166523 343 tablet_bootstrap.cc:492] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.168263 343 tablet_bootstrap.cc:654] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.173493 341 tablet_replica.cc:331] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:26.173785 341 raft_consensus.cc:2241] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.173969 341 raft_consensus.cc:2270] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.174641 341 ts_tablet_manager.cc:1905] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.179656 343 tablet_bootstrap.cc:492] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.179795 343 ts_tablet_manager.cc:1397] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250623 08:04:26.180488 343 raft_consensus.cc:357] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.180670 343 raft_consensus.cc:383] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.180756 343 raft_consensus.cc:738] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.182559 343 consensus_queue.cc:260] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.183656 343 ts_tablet_manager.cc:1428] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:26.183951 343 tablet_bootstrap.cc:492] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.185546 343 tablet_bootstrap.cc:654] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.190146 432 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "125745849172449ba4be8244d61fdcb3" instance_seqno: 1750665861404587) as {username='slave'} at 127.31.69.3:58369; Asking this server to re-register.
I20250623 08:04:26.195783 343 tablet_bootstrap.cc:492] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.195969 343 ts_tablet_manager.cc:1397] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250623 08:04:26.196264 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet b9d871507dcc46fc80f6e0ac739c6476 (table default.Table_d27db7e0fea34b5182353dee8ae23cac [id=816747bd5e004ae5964b7b885aa249fb]) successfully deleted
I20250623 08:04:26.196939 343 raft_consensus.cc:357] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.197187 343 raft_consensus.cc:383] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.197274 343 raft_consensus.cc:738] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.197553 343 consensus_queue.cc:260] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.197785 340 ts_tablet_manager.cc:1905] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.198305 453 raft_consensus.cc:491] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.198477 453 raft_consensus.cc:513] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.198868 339 tablet_bootstrap.cc:492] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.199005 339 ts_tablet_manager.cc:1397] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.217s user 0.026s sys 0.085s
I20250623 08:04:26.199195 453 leader_election.cc:290] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:26.199783 339 raft_consensus.cc:357] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.199960 339 raft_consensus.cc:383] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.200050 339 raft_consensus.cc:738] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.200268 339 consensus_queue.cc:260] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.200446 32701 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:26.200805 339 ts_tablet_manager.cc:1428] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:26.200925 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04a3ae58d26142d58c4df4f52d7e3e2c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:26.201114 32380 raft_consensus.cc:2466] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:26.200449 32655 raft_consensus.cc:3058] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.201112 339 tablet_bootstrap.cc:492] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.201821 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04a3ae58d26142d58c4df4f52d7e3e2c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:26.203195 32453 leader_election.cc:304] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:26.203256 432 ts_manager.cc:194] Registered new tserver with Master: 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:26.203531 32453 leader_election.cc:343] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:26.204977 339 tablet_bootstrap.cc:654] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.205407 32655 raft_consensus.cc:1273] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Refusing update from remote peer 59f7c614254f42009306b13337f06576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 08:04:26.206043 453 raft_consensus.cc:2802] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:26.206171 453 raft_consensus.cc:491] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:26.206269 453 raft_consensus.cc:3058] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.209188 453 raft_consensus.cc:513] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.209528 366 raft_consensus.cc:491] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.209661 366 raft_consensus.cc:513] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.210268 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04a3ae58d26142d58c4df4f52d7e3e2c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:26.210446 32380 raft_consensus.cc:3058] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.213274 32380 raft_consensus.cc:2466] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:26.213780 366 leader_election.cc:290] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:26.213765 32453 leader_election.cc:304] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:26.214422 343 ts_tablet_manager.cc:1428] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
W20250623 08:04:26.214859 32589 leader_election.cc:343] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 125784661a7b4e4ba6a710b950e004f8
I20250623 08:04:26.215126 366 raft_consensus.cc:491] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.215322 366 raft_consensus.cc:513] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.215909 366 leader_election.cc:290] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:26.216863 32589 leader_election.cc:343] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: b9d871507dcc46fc80f6e0ac739c6476
I20250623 08:04:26.217392 453 leader_election.cc:290] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:26.218456 343 tablet_bootstrap.cc:492] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.218686 453 consensus_queue.cc:1035] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:26.219534 453 raft_consensus.cc:2802] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:26.220189 453 raft_consensus.cc:695] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:26.220532 343 tablet_bootstrap.cc:654] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.220865 453 consensus_queue.cc:237] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.227486 339 tablet_bootstrap.cc:492] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.227658 339 ts_tablet_manager.cc:1397] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20250623 08:04:26.228487 339 raft_consensus.cc:357] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.228735 339 raft_consensus.cc:383] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.228825 339 raft_consensus.cc:738] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.229092 339 consensus_queue.cc:260] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.229729 339 ts_tablet_manager.cc:1428] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:26.230048 339 tablet_bootstrap.cc:492] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.231884 339 tablet_bootstrap.cc:654] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.237042 339 tablet_bootstrap.cc:492] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.237190 339 ts_tablet_manager.cc:1397] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250623 08:04:26.238063 339 raft_consensus.cc:357] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.238231 339 raft_consensus.cc:383] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.238330 339 raft_consensus.cc:738] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.239485 432 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7351aa31800245659edc9837e27daffd" instance_seqno: 1750665860475404) as {username='slave'} at 127.31.69.1:59097; Asking this server to re-register.
I20250623 08:04:26.239902 339 consensus_queue.cc:260] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.240469 339 ts_tablet_manager.cc:1428] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:26.240815 339 tablet_bootstrap.cc:492] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.241108 32426 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:26.242677 339 tablet_bootstrap.cc:654] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.243606 432 ts_manager.cc:194] Registered new tserver with Master: 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:26.245477 339 tablet_bootstrap.cc:492] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.245640 339 ts_tablet_manager.cc:1397] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250623 08:04:26.246420 339 raft_consensus.cc:357] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.246693 339 raft_consensus.cc:383] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.246804 339 raft_consensus.cc:738] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.247089 339 consensus_queue.cc:260] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.247581 339 ts_tablet_manager.cc:1428] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:26.247905 339 tablet_bootstrap.cc:492] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.249507 339 tablet_bootstrap.cc:654] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.250819 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet c7385d86316f4a3ab68a77d18080ec20 (table default.Table_465c441ac95145f1b498665ccedafe6c [id=24b0c7dcb97e4105b8edf674a0a90e84]) successfully deleted
W20250623 08:04:26.251294 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 8d8f38a6de1e45e7bed4abdced684c92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d8f38a6de1e45e7bed4abdced684c92 already in progress: creating tablet
W20250623 08:04:26.251520 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet d67a8f1f570741ebb886c4d136ca40d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d67a8f1f570741ebb886c4d136ca40d2 already in progress: creating tablet
W20250623 08:04:26.251739 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 1e47b10f46b8402e94354ebca573c194 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e47b10f46b8402e94354ebca573c194 already in progress: creating tablet
I20250623 08:04:26.251904 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet b6ad6cd629c4423ca997efb517cc975b (table default.Table_47264042125847ac997923572c293a3e [id=4214b1f0ca1744dc9b46d3f4d7618ba3]) successfully deleted
W20250623 08:04:26.252121 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 2795cb14e6db4116a4c9ee539bd4e82d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2795cb14e6db4116a4c9ee539bd4e82d already in progress: creating tablet
W20250623 08:04:26.252312 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet f0117cbb09dd4067836266c9d6a32adf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0117cbb09dd4067836266c9d6a32adf already in progress: creating tablet
I20250623 08:04:26.252394 339 tablet_bootstrap.cc:492] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.252527 339 ts_tablet_manager.cc:1397] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250623 08:04:26.252538 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]) successfully deleted
W20250623 08:04:26.252760 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet c6a97b34e386443fa19a9601621b5017 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6a97b34e386443fa19a9601621b5017 already in progress: creating tablet
W20250623 08:04:26.252964 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet d0ee5a4fe03848229524cc0105226d8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0ee5a4fe03848229524cc0105226d8c already in progress: creating tablet
I20250623 08:04:26.253131 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet ed5c81d9e56c4ee28546cedb9a579f31 (table default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]) successfully deleted
I20250623 08:04:26.253307 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet e90151394fb842498f4e780acab1f1aa (table default.Table_eefa8d393087436793f03c22b76ab152 [id=a1d4326f22fa44ba82cb376970f55ddc]) successfully deleted
I20250623 08:04:26.253266 339 raft_consensus.cc:357] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.253443 339 raft_consensus.cc:383] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.253482 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 125784661a7b4e4ba6a710b950e004f8 (table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]) successfully deleted
I20250623 08:04:26.253540 339 raft_consensus.cc:738] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.253768 339 consensus_queue.cc:260] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.254264 339 ts_tablet_manager.cc:1428] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:26.254613 339 tablet_bootstrap.cc:492] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.256543 339 tablet_bootstrap.cc:654] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.263123 587 raft_consensus.cc:491] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.263470 587 raft_consensus.cc:513] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.264441 587 leader_election.cc:290] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:26.272431 344 ts_tablet_manager.cc:1918] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:26.272656 344 log.cc:1199] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b6e29ab07df249748092a2f50a8c0273
I20250623 08:04:26.272586 32121 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_b7f7dd030adc46329a94b5de210eb2f9 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 2 got 0
I20250623 08:04:26.273185 344 ts_tablet_manager.cc:1939] T b6e29ab07df249748092a2f50a8c0273 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:26.274957 344 tablet_replica.cc:331] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:26.275151 344 raft_consensus.cc:2241] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.275228 32122 master_service.cc:948] ReplaceTablet: received request to replace tablet 32c82141edea4068b54755a85c6d7fb8 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:26.275339 344 raft_consensus.cc:2270] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.275694 32124 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet d4221d7500d1458db75926d7b274a654
I20250623 08:04:26.275954 344 ts_tablet_manager.cc:1905] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.278474 32124 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_b7f7dd030adc46329a94b5de210eb2f9 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 2 got 0
I20250623 08:04:26.281531 32124 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_ba70febc0949477ba25d45eafa4f7b43" } new_table_name: "default.Table_176ee9ee2de7444ca3c54ee24e3e5667" modify_external_catalogs: true
I20250623 08:04:26.283733 32121 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_baf5c1cd7eee47528cfdd1f4bf5f5d60" } new_table_name: "default.Table_4ee0e165259b4e2c88626b2d613149ca" modify_external_catalogs: true
W20250623 08:04:26.288379 32589 leader_election.cc:343] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: ed5c81d9e56c4ee28546cedb9a579f31
I20250623 08:04:26.288703 362 raft_consensus.cc:491] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.288928 362 raft_consensus.cc:513] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.289659 362 leader_election.cc:290] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:26.289714 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet d4221d7500d1458db75926d7b274a654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.290307 32619 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ca4957928fc4630aa8d8ca6110d7598 with delete_type TABLET_DATA_DELETED (replaced by tablet 67c44697ecab4dd9b43f544a602897ca) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.290566 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet b6ad6cd629c4423ca997efb517cc975b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.290935 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c77fd6e54904722acb2d1437c627ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0861107ab4624a05bd7a4d24b420c502) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.291141 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet b9d871507dcc46fc80f6e0ac739c6476 with delete_type TABLET_DATA_DELETED (replaced by tablet ca99c3b0cfec4aa481b620668b20947e) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.291431 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet d4221d7500d1458db75926d7b274a654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:26.295770 32314 leader_election.cc:343] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: a46c0a89fc014054a75ed198e824b5b6
I20250623 08:04:26.296177 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a46c0a89fc014054a75ed198e824b5b6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:26.297235 32314 leader_election.cc:343] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:26.297485 32314 leader_election.cc:304] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:26.297914 362 raft_consensus.cc:2747] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:26.298533 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet d4221d7500d1458db75926d7b274a654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.300347 342 tablet_bootstrap.cc:492] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:26.300526 342 ts_tablet_manager.cc:1397] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.239s user 0.048s sys 0.101s
I20250623 08:04:26.301376 342 raft_consensus.cc:357] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.301559 342 raft_consensus.cc:383] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.301653 342 raft_consensus.cc:738] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.301884 342 consensus_queue.cc:260] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.302500 342 ts_tablet_manager.cc:1428] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:26.302878 342 tablet_bootstrap.cc:492] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:26.303474 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b6e29ab07df249748092a2f50a8c0273 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]) successfully deleted
W20250623 08:04:26.303771 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa already in progress: creating tablet
W20250623 08:04:26.304001 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 67c44697ecab4dd9b43f544a602897ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67c44697ecab4dd9b43f544a602897ca already in progress: creating tablet
I20250623 08:04:26.304736 342 tablet_bootstrap.cc:654] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.307941 32126 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 35ab9f77a81a4490abb0d2c0c69cd90f
I20250623 08:04:26.308498 32126 catalog_manager.cc:6652] ReplaceTablet: tablet 35ab9f77a81a4490abb0d2c0c69cd90f deleted and replaced by tablet b1aa596950da49a898e3ed9bbad8c45e
I20250623 08:04:26.309094 341 ts_tablet_manager.cc:1918] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.309266 341 log.cc:1199] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/1e0c11ec768a43a48c6dc0f3dd974b71
I20250623 08:04:26.309684 341 ts_tablet_manager.cc:1939] T 1e0c11ec768a43a48c6dc0f3dd974b71 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:26.319137 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 105350us
I20250623 08:04:26.320148 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04a3ae58d26142d58c4df4f52d7e3e2c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:26.320381 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f with delete_type TABLET_DATA_DELETED (replaced by tablet b1aa596950da49a898e3ed9bbad8c45e) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.320434 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f with delete_type TABLET_DATA_DELETED (replaced by tablet b1aa596950da49a898e3ed9bbad8c45e) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.321058 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f with delete_type TABLET_DATA_DELETED (replaced by tablet b1aa596950da49a898e3ed9bbad8c45e) from {username='slave'} at 127.0.0.1:35526
W20250623 08:04:26.322265 32589 leader_election.cc:343] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 125784661a7b4e4ba6a710b950e004f8
I20250623 08:04:26.322512 32589 leader_election.cc:304] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:26.324819 587 raft_consensus.cc:2747] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:26.325398 32589 leader_election.cc:343] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: b9d871507dcc46fc80f6e0ac739c6476
I20250623 08:04:26.325600 32589 leader_election.cc:304] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:26.326169 587 raft_consensus.cc:2747] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:26.326664 32589 leader_election.cc:343] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: ed5c81d9e56c4ee28546cedb9a579f31
I20250623 08:04:26.326835 32589 leader_election.cc:304] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:26.327382 587 raft_consensus.cc:2747] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:26.327847 32453 leader_election.cc:343] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:26.329689 432 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "59f7c614254f42009306b13337f06576" instance_seqno: 1750665860944300) as {username='slave'} at 127.31.69.2:60995; Asking this server to re-register.
I20250623 08:04:26.331442 603 tablet.cc:1722] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:26.332126 32566 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:26.336184 432 ts_manager.cc:194] Registered new tserver with Master: 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:26.339478 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 1e0c11ec768a43a48c6dc0f3dd974b71 (table default.Table_4d4535cab44545839eb3a559385e395a [id=bb849e86d7084959830c68207fb99061]) successfully deleted
I20250623 08:04:26.340057 343 tablet_bootstrap.cc:492] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.340240 343 ts_tablet_manager.cc:1397] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.126s user 0.005s sys 0.010s
I20250623 08:04:26.341216 343 raft_consensus.cc:357] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.341425 343 raft_consensus.cc:383] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.341531 343 raft_consensus.cc:738] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.341812 343 consensus_queue.cc:260] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.342386 343 ts_tablet_manager.cc:1428] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:26.342684 343 tablet_bootstrap.cc:492] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.342689 344 ts_tablet_manager.cc:1918] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.342832 344 log.cc:1199] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/0c77fd6e54904722acb2d1437c627ba8
I20250623 08:04:26.343248 344 ts_tablet_manager.cc:1939] T 0c77fd6e54904722acb2d1437c627ba8 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:26.352789 340 ts_tablet_manager.cc:1918] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.354348 343 tablet_bootstrap.cc:654] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.355569 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 0c77fd6e54904722acb2d1437c627ba8 (table default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]) successfully deleted
W20250623 08:04:26.355938 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet d4221d7500d1458db75926d7b274a654 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4221d7500d1458db75926d7b274a654 already in progress: creating tablet
I20250623 08:04:26.363579 343 tablet_bootstrap.cc:492] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.363768 343 ts_tablet_manager.cc:1397] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250623 08:04:26.364553 343 raft_consensus.cc:357] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:26.364724 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 35ab9f77a81a4490abb0d2c0c69cd90f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35ab9f77a81a4490abb0d2c0c69cd90f already in progress: creating tablet
I20250623 08:04:26.364769 343 raft_consensus.cc:383] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.364835 343 raft_consensus.cc:738] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.367982 457 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:26.371104 343 consensus_queue.cc:260] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.310698 341 tablet_replica.cc:331] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:26.372025 341 raft_consensus.cc:2241] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.372344 341 raft_consensus.cc:2270] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:26.378804 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f9e0d1b85a94ae9ae86e53688a55db5 has no consensus state
I20250623 08:04:26.380964 32123 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ea37269e0aaf457bac98f5ffedb8c981
I20250623 08:04:26.381181 32123 master_service.cc:948] ReplaceTablet: received request to replace tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b from {username='slave'} at 127.0.0.1:50426
W20250623 08:04:26.381757 32124 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ea81642050141a9be84f151a5785abc consensus state has no leader
I20250623 08:04:26.381968 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet 5629d11c44e241d598f3e519b2fe1a44 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:26.365006 362 raft_consensus.cc:491] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.383138 362 raft_consensus.cc:513] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:26.383374 32121 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3d32798168345edb9898751d083057c has no consensus state
I20250623 08:04:26.383987 362 leader_election.cc:290] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:26.384680 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_32f32ccf1e554c3aa3e331b0b9de44d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.384781 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d67a8f1f570741ebb886c4d136ca40d2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:26.385308 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32f32ccf1e554c3aa3e331b0b9de44d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:26.386051 32314 leader_election.cc:343] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:26.366060 604 tablet.cc:1722] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:26.386565 32128 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_509603863ff145f19f99bc7d6743f562" } new_table_name: "default.Table_6648aba827f24ba29ee9d94ee7c96089" modify_external_catalogs: true
I20250623 08:04:26.388049 340 log.cc:1199] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/35428260e05e447ebf88ea41e63cdd02
I20250623 08:04:26.388581 340 ts_tablet_manager.cc:1939] T 35428260e05e447ebf88ea41e63cdd02 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.389406 32130 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_86a7bf5fc79045769686a82168ac66da" } new_table_name: "default.Table_3540dde86f8442a89a59232aa4b53612" modify_external_catalogs: true
I20250623 08:04:26.372852 343 ts_tablet_manager.cc:1428] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250623 08:04:26.390112 343 tablet_bootstrap.cc:492] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.385546 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d67a8f1f570741ebb886c4d136ca40d2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:26.391152 32655 raft_consensus.cc:2466] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:26.391726 32314 leader_election.cc:304] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:26.392988 32123 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_5e8e88b2237f435595d619af47384b4e" } modify_external_catalogs: true
I20250623 08:04:26.393183 32123 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_5e8e88b2237f435595d619af47384b4e" } modify_external_catalogs: true
I20250623 08:04:26.393306 362 raft_consensus.cc:2802] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:26.393424 362 raft_consensus.cc:491] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:26.393545 362 raft_consensus.cc:3058] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.395875 343 tablet_bootstrap.cc:654] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.396499 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_bc69503004f74bf59fd961b4f209baa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:26.397123 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc69503004f74bf59fd961b4f209baa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.399699 362 raft_consensus.cc:513] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.401561 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d67a8f1f570741ebb886c4d136ca40d2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:26.402478 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d67a8f1f570741ebb886c4d136ca40d2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:26.402717 32655 raft_consensus.cc:3058] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.406149 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 35428260e05e447ebf88ea41e63cdd02 (table default.Table_c5dbc5b105b8416c90d3792d29568b83 [id=a1839631f0c64b16a3f0411b514f93a2]) successfully deleted
I20250623 08:04:26.406226 607 tablet.cc:1722] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:26.407629 32655 raft_consensus.cc:2466] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
W20250623 08:04:26.407935 32314 leader_election.cc:343] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:26.409036 32314 leader_election.cc:304] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
W20250623 08:04:26.409200 32128 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a25bdc9b7e48467fba3771365e58bcbc consensus state has no leader
I20250623 08:04:26.410984 453 raft_consensus.cc:491] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.411168 453 raft_consensus.cc:513] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.411798 453 leader_election.cc:290] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:26.412029 362 leader_election.cc:290] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:26.412703 362 raft_consensus.cc:2802] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:26.412770 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ca4957928fc4630aa8d8ca6110d7598" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:26.413028 32655 raft_consensus.cc:2466] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:26.414832 32453 leader_election.cc:304] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:26.415091 362 raft_consensus.cc:695] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:26.417537 32128 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_92e23e1081544f8c9b99f14654981494" } modify_external_catalogs: true
I20250623 08:04:26.417776 32128 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_92e23e1081544f8c9b99f14654981494" } modify_external_catalogs: true
I20250623 08:04:26.418267 362 consensus_queue.cc:237] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.419188 343 tablet_bootstrap.cc:492] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.419413 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_d44a2132435f43e7847bfa349645a605"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.419909 453 raft_consensus.cc:2802] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:26.420058 453 raft_consensus.cc:491] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 08:04:26.420113 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d44a2132435f43e7847bfa349645a605 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.420166 453 raft_consensus.cc:3058] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.424331 453 raft_consensus.cc:513] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.425390 453 leader_election.cc:290] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:26.426177 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ca4957928fc4630aa8d8ca6110d7598" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:26.427090 32655 raft_consensus.cc:3058] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.430081 342 tablet_bootstrap.cc:492] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:26.430240 342 ts_tablet_manager.cc:1397] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.127s user 0.000s sys 0.045s
I20250623 08:04:26.431537 32655 raft_consensus.cc:2466] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:26.431047 342 raft_consensus.cc:357] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.431736 342 raft_consensus.cc:383] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.431813 342 raft_consensus.cc:738] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.432039 342 consensus_queue.cc:260] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.432286 32453 leader_election.cc:304] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:26.433391 32130 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fe30079045d437aa61bb5bfbfb96133 consensus state has no leader
I20250623 08:04:26.433732 32130 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030" } modify_external_catalogs: true
I20250623 08:04:26.433821 343 ts_tablet_manager.cc:1397] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250623 08:04:26.433905 32130 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030" } modify_external_catalogs: true
I20250623 08:04:26.432514 342 ts_tablet_manager.cc:1428] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:26.435379 342 tablet_bootstrap.cc:492] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:26.436379 453 raft_consensus.cc:2802] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:26.436575 453 raft_consensus.cc:695] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:26.436908 453 consensus_queue.cc:237] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.442677 343 raft_consensus.cc:357] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.442943 343 raft_consensus.cc:383] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.443045 343 raft_consensus.cc:738] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.443296 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_244bcdc8d69b4f61a337dac403ffe6cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.443341 343 consensus_queue.cc:260] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:26.443856 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_244bcdc8d69b4f61a337dac403ffe6cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.445021 32128 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet b1aa596950da49a898e3ed9bbad8c45e
I20250623 08:04:26.445230 32128 master_service.cc:948] ReplaceTablet: received request to replace tablet 7da84969a6464b5caeaf47af0cbaaa4d from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:26.445986 342 tablet_bootstrap.cc:654] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.448592 32126 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_b7f7dd030adc46329a94b5de210eb2f9 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 2 got 0
I20250623 08:04:26.451691 32125 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_b7f7dd030adc46329a94b5de210eb2f9 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 2 got 0
I20250623 08:04:26.451817 343 ts_tablet_manager.cc:1428] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250623 08:04:26.452095 343 tablet_bootstrap.cc:492] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.453847 32127 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_b7f7dd030adc46329a94b5de210eb2f9 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 2 got 0
I20250623 08:04:26.455600 341 ts_tablet_manager.cc:1905] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.459462 453 raft_consensus.cc:491] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.459642 453 raft_consensus.cc:513] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.460278 453 leader_election.cc:290] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:26.460968 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c1596e80225041099fb513ce477eae33" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:26.461164 32655 raft_consensus.cc:2466] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:26.460573 339 tablet_bootstrap.cc:492] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.461697 339 ts_tablet_manager.cc:1397] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.207s user 0.055s sys 0.108s
I20250623 08:04:26.462368 343 tablet_bootstrap.cc:654] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.463291 32354 tablet_service.cc:1468] Processing CreateTablet for tablet adc3cf4355c24b7395fcb22af8520346 (DEFAULT_TABLE table=default.Table_8ef30265c1894796843272f538aaa79e [id=ad13fb2c08c448bebe3e65b0143d66bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.463292 32494 tablet_service.cc:1468] Processing CreateTablet for tablet adc3cf4355c24b7395fcb22af8520346 (DEFAULT_TABLE table=default.Table_8ef30265c1894796843272f538aaa79e [id=ad13fb2c08c448bebe3e65b0143d66bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.463443 32636 tablet_service.cc:1468] Processing CreateTablet for tablet adc3cf4355c24b7395fcb22af8520346 (DEFAULT_TABLE table=default.Table_8ef30265c1894796843272f538aaa79e [id=ad13fb2c08c448bebe3e65b0143d66bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.463677 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc3cf4355c24b7395fcb22af8520346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.463696 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc3cf4355c24b7395fcb22af8520346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.464876 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 0545daa6fba846adbc573a63d8dbcd66 (DEFAULT_TABLE table=default.Table_e69f9fec5b604bca8658b2124145f92e [id=96cf904f9ea64544ae6f662208874f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.466176 32630 tablet_service.cc:1468] Processing CreateTablet for tablet c0f87fd62f1545089d00757f8214593a (DEFAULT_TABLE table=default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.467392 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 283bebc7063d437ba3e541d7dee3c0b5 (DEFAULT_TABLE table=default.Table_3878732ceac64ef6bc3f0d170ffbe12a [id=c8fe0110bcd04cc2a14886d4f3a81dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.467965 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 35428260e05e447ebf88ea41e63cdd02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.468251 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet 04a3ae58d26142d58c4df4f52d7e3e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.468477 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet 69e772df590747ff841be5f218e55acf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.468677 32498 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ca4957928fc4630aa8d8ca6110d7598 with delete_type TABLET_DATA_DELETED (replaced by tablet 67c44697ecab4dd9b43f544a602897ca) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.469058 32619 tablet_service.cc:1468] Processing CreateTablet for tablet f44fd3c67185484e93450682364dc8a8 (DEFAULT_TABLE table=default.Table_e2502c251f3c4a7f8caa4faab352ebda [id=664b85653cd341a6a6afe7d04fb9c2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.469256 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 0545daa6fba846adbc573a63d8dbcd66 (DEFAULT_TABLE table=default.Table_e69f9fec5b604bca8658b2124145f92e [id=96cf904f9ea64544ae6f662208874f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.469599 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0545daa6fba846adbc573a63d8dbcd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.470330 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 2060a643d0da47efa868f52bdb331b1f (DEFAULT_TABLE table=default.Table_92ca14ac8b9647b0bd3b515217b30323 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.471801 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 358cb6ac24414bdfbc32155e732f9ac8 (DEFAULT_TABLE table=default.Table_2b1942032adc43d78d1f9e83417b42eb [id=b827ea08f0b14c9aa2b2c3d02d15f150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.472723 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet d67a8f1f570741ebb886c4d136ca40d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.473057 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 69e772df590747ff841be5f218e55acf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.473109 32495 tablet_service.cc:1468] Processing CreateTablet for tablet c0f87fd62f1545089d00757f8214593a (DEFAULT_TABLE table=default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.473462 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f87fd62f1545089d00757f8214593a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.473923 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 0545daa6fba846adbc573a63d8dbcd66 (DEFAULT_TABLE table=default.Table_e69f9fec5b604bca8658b2124145f92e [id=96cf904f9ea64544ae6f662208874f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.474241 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0545daa6fba846adbc573a63d8dbcd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.473714 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_2c99afea2b6e4963985dab32b545779e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:26.475286 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c99afea2b6e4963985dab32b545779e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.478906 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 2eb99713f9214e6488d5c6e140ac263c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.479050 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 283bebc7063d437ba3e541d7dee3c0b5 (DEFAULT_TABLE table=default.Table_3878732ceac64ef6bc3f0d170ffbe12a [id=c8fe0110bcd04cc2a14886d4f3a81dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.479260 32353 tablet_service.cc:1468] Processing CreateTablet for tablet c0f87fd62f1545089d00757f8214593a (DEFAULT_TABLE table=default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.479406 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283bebc7063d437ba3e541d7dee3c0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.479599 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f87fd62f1545089d00757f8214593a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.482870 344 tablet_replica.cc:331] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:26.483099 344 raft_consensus.cc:2241] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.483256 344 raft_consensus.cc:2270] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.483808 344 ts_tablet_manager.cc:1905] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.485106 32129 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_4c5e4af6e0da4344a426fc6bcfafdc97" } modify_external_catalogs: true
I20250623 08:04:26.485286 32129 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_4c5e4af6e0da4344a426fc6bcfafdc97" } modify_external_catalogs: true
I20250623 08:04:26.486508 587 raft_consensus.cc:491] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.486671 587 raft_consensus.cc:513] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.486879 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.487097 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c77fd6e54904722acb2d1437c627ba8 with delete_type TABLET_DATA_DELETED (replaced by tablet 0861107ab4624a05bd7a4d24b420c502) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.487361 587 leader_election.cc:290] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:26.487426 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 283bebc7063d437ba3e541d7dee3c0b5 (DEFAULT_TABLE table=default.Table_3878732ceac64ef6bc3f0d170ffbe12a [id=c8fe0110bcd04cc2a14886d4f3a81dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.487800 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283bebc7063d437ba3e541d7dee3c0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.488044 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_cdabecebee2e4010a7325455f785d002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.488319 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet a46c0a89fc014054a75ed198e824b5b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 228375592aee48a1aa21e1d16962b663) from {username='slave'} at 127.0.0.1:35526
W20250623 08:04:26.488591 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdabecebee2e4010a7325455f785d002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.488664 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c77fd6e54904722acb2d1437c627ba8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:26.489004 32379 raft_consensus.cc:2466] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:26.489514 32589 leader_election.cc:304] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:26.489801 587 raft_consensus.cc:2802] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:26.489907 587 raft_consensus.cc:491] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:26.489981 587 raft_consensus.cc:3058] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.491025 32356 tablet_service.cc:1468] Processing CreateTablet for tablet f44fd3c67185484e93450682364dc8a8 (DEFAULT_TABLE table=default.Table_e2502c251f3c4a7f8caa4faab352ebda [id=664b85653cd341a6a6afe7d04fb9c2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.491367 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f44fd3c67185484e93450682364dc8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.492846 587 raft_consensus.cc:513] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.493454 587 leader_election.cc:290] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:26.494257 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c77fd6e54904722acb2d1437c627ba8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:26.494441 32380 raft_consensus.cc:3058] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.495047 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 2060a643d0da47efa868f52bdb331b1f (DEFAULT_TABLE table=default.Table_92ca14ac8b9647b0bd3b515217b30323 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.495407 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2060a643d0da47efa868f52bdb331b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.497340 32380 raft_consensus.cc:2466] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:26.497810 32589 leader_election.cc:304] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:26.498142 587 raft_consensus.cc:2802] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:26.498267 587 raft_consensus.cc:695] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:26.498554 587 consensus_queue.cc:237] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.498639 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 358cb6ac24414bdfbc32155e732f9ac8 (DEFAULT_TABLE table=default.Table_2b1942032adc43d78d1f9e83417b42eb [id=b827ea08f0b14c9aa2b2c3d02d15f150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.499056 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358cb6ac24414bdfbc32155e732f9ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.500651 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (DEFAULT_TABLE table=default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.500998 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9e0d1b85a94ae9ae86e53688a55db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.504509 32453 leader_election.cc:304] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:26.504904 32589 leader_election.cc:343] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 0c77fd6e54904722acb2d1437c627ba8
W20250623 08:04:26.505544 32589 leader_election.cc:343] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 0c77fd6e54904722acb2d1437c627ba8
I20250623 08:04:26.505609 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 2060a643d0da47efa868f52bdb331b1f (DEFAULT_TABLE table=default.Table_92ca14ac8b9647b0bd3b515217b30323 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.506001 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2060a643d0da47efa868f52bdb331b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.508010 453 raft_consensus.cc:2802] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:26.508109 453 raft_consensus.cc:491] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:26.508209 453 raft_consensus.cc:3058] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.511497 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 358cb6ac24414bdfbc32155e732f9ac8 (DEFAULT_TABLE table=default.Table_2b1942032adc43d78d1f9e83417b42eb [id=b827ea08f0b14c9aa2b2c3d02d15f150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.511924 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358cb6ac24414bdfbc32155e732f9ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.514492 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc3cf4355c24b7395fcb22af8520346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.516319 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358cb6ac24414bdfbc32155e732f9ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.518918 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0545daa6fba846adbc573a63d8dbcd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.521693 341 ts_tablet_manager.cc:1918] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.521832 341 log.cc:1199] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/ed5c81d9e56c4ee28546cedb9a579f31
I20250623 08:04:26.522204 341 ts_tablet_manager.cc:1939] T ed5c81d9e56c4ee28546cedb9a579f31 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:26.522910 32619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f44fd3c67185484e93450682364dc8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.462594 339 raft_consensus.cc:357] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.526888 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0f87fd62f1545089d00757f8214593a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.527354 339 raft_consensus.cc:383] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.527469 339 raft_consensus.cc:738] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.527694 339 consensus_queue.cc:260] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.528126 339 ts_tablet_manager.cc:1428] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.066s user 0.001s sys 0.001s
I20250623 08:04:26.528355 339 tablet_bootstrap.cc:492] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.529577 339 tablet_bootstrap.cc:654] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.530898 576 catalog_manager.cc:1261] Loaded cluster ID: bde910f56f35432e94ef19d006edb93c
I20250623 08:04:26.530911 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 283bebc7063d437ba3e541d7dee3c0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.530997 576 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: loading cluster ID for follower catalog manager: success
I20250623 08:04:26.531303 339 tablet_bootstrap.cc:492] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.531443 339 ts_tablet_manager.cc:1397] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:26.532196 339 raft_consensus.cc:357] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.532369 339 raft_consensus.cc:383] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.532457 339 raft_consensus.cc:738] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.532650 339 consensus_queue.cc:260] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.535212 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2060a643d0da47efa868f52bdb331b1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:26.539284 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 125432us
I20250623 08:04:26.539577 343 tablet_bootstrap.cc:492] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.539745 343 ts_tablet_manager.cc:1397] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.088s user 0.002s sys 0.001s
I20250623 08:04:26.540555 343 raft_consensus.cc:357] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.540763 343 raft_consensus.cc:383] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.540844 343 raft_consensus.cc:738] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.541072 343 consensus_queue.cc:260] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.542649 343 ts_tablet_manager.cc:1428] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:26.542979 343 tablet_bootstrap.cc:492] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.543936 339 ts_tablet_manager.cc:1428] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250623 08:04:26.544595 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c1596e80225041099fb513ce477eae33" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:26.546885 32381 raft_consensus.cc:2466] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:26.547032 339 tablet_bootstrap.cc:492] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.546833 340 tablet_replica.cc:331] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.552433 32491 tablet_service.cc:1468] Processing CreateTablet for tablet f44fd3c67185484e93450682364dc8a8 (DEFAULT_TABLE table=default.Table_e2502c251f3c4a7f8caa4faab352ebda [id=664b85653cd341a6a6afe7d04fb9c2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.552563 32356 tablet_service.cc:1468] Processing CreateTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 (DEFAULT_TABLE table=default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.552942 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 38ecd4c243b840f4b66d8e01078dca64 (DEFAULT_TABLE table=default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.553403 32348 tablet_service.cc:1468] Processing CreateTablet for tablet f3d32798168345edb9898751d083057c (DEFAULT_TABLE table=default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.553754 340 raft_consensus.cc:2241] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:26.553807 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa already in progress: creating tablet
W20250623 08:04:26.553933 32453 leader_election.cc:343] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 5ca4957928fc4630aa8d8ca6110d7598
W20250623 08:04:26.554069 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 67c44697ecab4dd9b43f544a602897ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67c44697ecab4dd9b43f544a602897ca already in progress: creating tablet
I20250623 08:04:26.554183 340 raft_consensus.cc:2270] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:26.554438 32453 leader_election.cc:343] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 5ca4957928fc4630aa8d8ca6110d7598
I20250623 08:04:26.554867 340 ts_tablet_manager.cc:1905] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.555787 339 tablet_bootstrap.cc:654] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.556602 344 ts_tablet_manager.cc:1918] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.556742 344 log.cc:1199] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/35428260e05e447ebf88ea41e63cdd02
I20250623 08:04:26.557108 344 ts_tablet_manager.cc:1939] T 35428260e05e447ebf88ea41e63cdd02 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:26.557574 344 tablet_replica.cc:331] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:26.557765 344 raft_consensus.cc:2241] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:26.558022 344 pending_rounds.cc:70] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:26.558187 344 pending_rounds.cc:77] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727388049457152 op_type: NO_OP noop_request { }
I20250623 08:04:26.559679 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cb66c7a273a412db3ae77af9e30d036. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.563650 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38ecd4c243b840f4b66d8e01078dca64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.564098 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3d32798168345edb9898751d083057c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.566845 342 tablet_bootstrap.cc:492] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:26.567019 342 ts_tablet_manager.cc:1397] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.132s user 0.003s sys 0.002s
I20250623 08:04:26.568961 32347 tablet_service.cc:1468] Processing CreateTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 (DEFAULT_TABLE table=default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.569355 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea702abb31fb41498ab61bbb05b1f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.574961 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 2ecf4af9e476495683bd63cac20d4672 (DEFAULT_TABLE table=default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.575152 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (DEFAULT_TABLE table=default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.575346 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ecf4af9e476495683bd63cac20d4672. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.575763 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 35428260e05e447ebf88ea41e63cdd02 (table default.Table_c5dbc5b105b8416c90d3792d29568b83 [id=a1839631f0c64b16a3f0411b514f93a2]) successfully deleted
I20250623 08:04:26.576561 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f44fd3c67185484e93450682364dc8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.577828 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 20e7a78cfc384033810603316b9e66bf (DEFAULT_TABLE table=default.Table_7987e1a82b644bd69724bc61cd84445a [id=96f6aedcc9704226b4b14e5548a35ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.578394 32497 tablet_service.cc:1468] Processing CreateTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 (DEFAULT_TABLE table=default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.578999 32488 tablet_service.cc:1468] Processing CreateTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a (DEFAULT_TABLE table=default.Table_63662f63088548e29e6db7281e66691e [id=ec1f9e35e20c47fcbe2d4b80c7abeabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.579277 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 38ecd4c243b840f4b66d8e01078dca64 (DEFAULT_TABLE table=default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.579632 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 2ecf4af9e476495683bd63cac20d4672 (DEFAULT_TABLE table=default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.580137 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a (DEFAULT_TABLE table=default.Table_63662f63088548e29e6db7281e66691e [id=ec1f9e35e20c47fcbe2d4b80c7abeabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.580477 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965f1f0a107b4653b6aac01e0d8e259a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.580919 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 7da84969a6464b5caeaf47af0cbaaa4d
I20250623 08:04:26.581323 32128 catalog_manager.cc:6652] ReplaceTablet: tablet 7da84969a6464b5caeaf47af0cbaaa4d deleted and replaced by tablet 136e482995a24c5daec55a701f3091fa
I20250623 08:04:26.582158 32495 tablet_service.cc:1468] Processing CreateTablet for tablet f3d32798168345edb9898751d083057c (DEFAULT_TABLE table=default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.558352 344 raft_consensus.cc:2887] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:26.584241 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 (DEFAULT_TABLE table=default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.585072 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9e0d1b85a94ae9ae86e53688a55db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.592036 453 raft_consensus.cc:513] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.593984 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 32c82141edea4068b54755a85c6d7fb8
I20250623 08:04:26.594473 32122 catalog_manager.cc:6652] ReplaceTablet: tablet 32c82141edea4068b54755a85c6d7fb8 deleted and replaced by tablet 162308da898d4dde8ab5b1ab01d0f393
I20250623 08:04:26.594529 342 raft_consensus.cc:357] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.594722 342 raft_consensus.cc:383] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.594835 342 raft_consensus.cc:738] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.594832 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_f81a1165815543b781251c1ecfd06779"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.595060 342 consensus_queue.cc:260] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:26.595341 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f81a1165815543b781251c1ecfd06779 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.595371 32130 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 19536c16ff32460e9db571e4ce411bab
I20250623 08:04:26.595580 342 ts_tablet_manager.cc:1428] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250623 08:04:26.595866 342 tablet_bootstrap.cc:492] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:26.596125 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e7a78cfc384033810603316b9e66bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.596048 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_3ef2e86e99164cce8e882e6954e2ec17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.596513 32124 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 5629d11c44e241d598f3e519b2fe1a44
I20250623 08:04:26.596647 32127 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: default.Table_b7f7dd030adc46329a94b5de210eb2f9 [id=cb2c6ba697ca494c8eb60b65624a0517] alter complete (version 2)
I20250623 08:04:26.596715 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38ecd4c243b840f4b66d8e01078dca64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.596979 32124 catalog_manager.cc:6652] ReplaceTablet: tablet 5629d11c44e241d598f3e519b2fe1a44 deleted and replaced by tablet 205da880685b49b3b8669f73dc2c563b
I20250623 08:04:26.597295 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_fe75261182e243ebbbcfbf6e636ba134"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.597698 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_4550e74cc6e845719c1466d3acd1b723"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.597963 32123 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet ce1f3ef3690c4837b95b105733884f1f
I20250623 08:04:26.598079 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea702abb31fb41498ab61bbb05b1f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:26.598119 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4550e74cc6e845719c1466d3acd1b723 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.598285 32356 tablet_service.cc:1515] Processing DeleteTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d with delete_type TABLET_DATA_DELETED (replaced by tablet 136e482995a24c5daec55a701f3091fa) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.598613 32123 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_7b31ef6cd54f4eec82e08a3e592b1dcf" } new_table_name: "default.Table_ec93cb7b48b247d08d3832d2589c93b7" modify_external_catalogs: true
I20250623 08:04:26.598950 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ecf4af9e476495683bd63cac20d4672. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.599498 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cb66c7a273a412db3ae77af9e30d036. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.600549 32129 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 16d0fcc215934ae8a5bb57cd23b372e5
I20250623 08:04:26.600812 32129 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_176ee9ee2de7444ca3c54ee24e3e5667" } modify_external_catalogs: true
I20250623 08:04:26.600862 32128 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_32f32ccf1e554c3aa3e331b0b9de44d1" } new_table_name: "default.Table_52d445f76863453eb994aa3b0736a5f0" modify_external_catalogs: true
I20250623 08:04:26.600983 32129 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_176ee9ee2de7444ca3c54ee24e3e5667" } modify_external_catalogs: true
I20250623 08:04:26.601023 32487 tablet_service.cc:1515] Processing DeleteTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d with delete_type TABLET_DATA_DELETED (replaced by tablet 136e482995a24c5daec55a701f3091fa) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.601408 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 20e7a78cfc384033810603316b9e66bf (DEFAULT_TABLE table=default.Table_7987e1a82b644bd69724bc61cd84445a [id=96f6aedcc9704226b4b14e5548a35ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.601792 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e7a78cfc384033810603316b9e66bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.602311 32126 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_92ca14ac8b9647b0bd3b515217b30323" } new_table_name: "default.Table_fa02183517ad4115af0d9fd8ce2cd5d6" modify_external_catalogs: true
I20250623 08:04:26.604768 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 32c82141edea4068b54755a85c6d7fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 162308da898d4dde8ab5b1ab01d0f393) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.605546 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 19536c16ff32460e9db571e4ce411bab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.606195 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 with delete_type TABLET_DATA_DELETED (replaced by tablet 205da880685b49b3b8669f73dc2c563b) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.606995 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet ce1f3ef3690c4837b95b105733884f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35526
W20250623 08:04:26.596536 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ef2e86e99164cce8e882e6954e2ec17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.608112 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965f1f0a107b4653b6aac01e0d8e259a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:26.597719 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe75261182e243ebbbcfbf6e636ba134 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.608848 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3d32798168345edb9898751d083057c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.609792 339 tablet_bootstrap.cc:492] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.609937 339 ts_tablet_manager.cc:1397] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.000s
I20250623 08:04:26.610682 339 raft_consensus.cc:357] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.610903 339 raft_consensus.cc:383] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.610980 339 raft_consensus.cc:738] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.611032 32125 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8f909d419b67443f82c2c2d381c84a99" } new_table_name: "default.Table_a22f57b4aaa54648ab0a7c73aa2e887c" modify_external_catalogs: true
I20250623 08:04:26.611320 339 consensus_queue.cc:260] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.611835 339 ts_tablet_manager.cc:1428] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:26.612139 339 tablet_bootstrap.cc:492] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.612586 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_6e40e423312e4f9e95174f277b8897ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:26.613123 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e40e423312e4f9e95174f277b8897ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.613844 339 tablet_bootstrap.cc:654] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.613890 344 raft_consensus.cc:2270] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.614614 344 ts_tablet_manager.cc:1905] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.614930 340 ts_tablet_manager.cc:1918] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.615072 340 log.cc:1199] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/04a3ae58d26142d58c4df4f52d7e3e2c
I20250623 08:04:26.617599 340 ts_tablet_manager.cc:1939] T 04a3ae58d26142d58c4df4f52d7e3e2c P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.618518 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 04a3ae58d26142d58c4df4f52d7e3e2c (table default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]) successfully deleted
W20250623 08:04:26.619169 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet d4221d7500d1458db75926d7b274a654 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4221d7500d1458db75926d7b274a654 already in progress: creating tablet
I20250623 08:04:26.619988 340 tablet_replica.cc:331] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.620203 340 raft_consensus.cc:2241] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:26.620450 340 pending_rounds.cc:70] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:26.620543 340 pending_rounds.cc:77] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727388938350592 op_type: NO_OP noop_request { }
I20250623 08:04:26.620734 340 raft_consensus.cc:2887] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:26.620867 340 raft_consensus.cc:2270] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.621742 340 ts_tablet_manager.cc:1905] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.623951 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 32c82141edea4068b54755a85c6d7fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 162308da898d4dde8ab5b1ab01d0f393) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.627511 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 19536c16ff32460e9db571e4ce411bab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.628402 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 with delete_type TABLET_DATA_DELETED (replaced by tablet 205da880685b49b3b8669f73dc2c563b) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.629148 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet ce1f3ef3690c4837b95b105733884f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.629444 32487 tablet_service.cc:1515] Processing DeleteTablet for tablet 2eb99713f9214e6488d5c6e140ac263c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.635509 32487 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.637151 340 ts_tablet_manager.cc:1918] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:26.638350 340 log.cc:1199] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/d67a8f1f570741ebb886c4d136ca40d2
I20250623 08:04:26.638852 340 ts_tablet_manager.cc:1939] T d67a8f1f570741ebb886c4d136ca40d2 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.642489 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet d67a8f1f570741ebb886c4d136ca40d2 (table default.Table_be622804eb114901b245383926c3afec [id=b020e49a601748e9bcbf397e235dc596]) successfully deleted
I20250623 08:04:26.643164 340 tablet_replica.cc:331] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.643383 340 raft_consensus.cc:2241] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.643554 340 raft_consensus.cc:2270] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.644976 340 ts_tablet_manager.cc:1905] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.645712 32122 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c1bef462c12a418aa2d691eca2e06677" } new_table_name: "default.Table_50f04b0656c9469fb7bb4beee4b9268c" modify_external_catalogs: true
I20250623 08:04:26.649000 342 tablet_bootstrap.cc:654] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.654075 32487 tablet_service.cc:1515] Processing DeleteTablet for tablet d67a8f1f570741ebb886c4d136ca40d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.660001 340 ts_tablet_manager.cc:1918] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.660152 340 log.cc:1199] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/69e772df590747ff841be5f218e55acf
I20250623 08:04:26.660465 340 ts_tablet_manager.cc:1939] T 69e772df590747ff841be5f218e55acf P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:26.661023 32123 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ac9f38877b24d619c280a1aaa0b7b08 consensus state has no leader
I20250623 08:04:26.661475 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 69e772df590747ff841be5f218e55acf (table default.Table_c3e871b94f3944d6b2b75626c3af94ec [id=a61a71f32ae444338b041f6e0a9fa286]) successfully deleted
I20250623 08:04:26.663717 340 tablet_replica.cc:331] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.663909 340 raft_consensus.cc:2241] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.663884 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_cb9f889d49ac4ff8b2cfdbc50b28c547"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.664083 340 raft_consensus.cc:2270] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:26.664491 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb9f889d49ac4ff8b2cfdbc50b28c547 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.665500 340 ts_tablet_manager.cc:1905] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.669256 610 raft_consensus.cc:491] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.669440 610 raft_consensus.cc:513] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.670171 610 leader_election.cc:290] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:26.672698 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c4dec1a48f84fb3bd7bbd757f7b0c1b" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:26.675118 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c1596e80225041099fb513ce477eae33" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:26.675365 32381 raft_consensus.cc:3058] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:26.677321 340 ts_tablet_manager.cc:1918] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.678437 32381 raft_consensus.cc:2466] T c1596e80225041099fb513ce477eae33 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:26.677433 340 log.cc:1199] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/2eb99713f9214e6488d5c6e140ac263c
I20250623 08:04:26.680830 340 ts_tablet_manager.cc:1939] T 2eb99713f9214e6488d5c6e140ac263c P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.681737 340 tablet_replica.cc:331] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.681869 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 2eb99713f9214e6488d5c6e140ac263c (table default.Table_2807052f51c24614b4068e8113db2abd [id=9e90ae34db704c408759a60271e0bf82]) successfully deleted
I20250623 08:04:26.681953 340 raft_consensus.cc:2241] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.682116 340 raft_consensus.cc:2270] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.683859 32520 raft_consensus.cc:2466] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:26.683878 32453 leader_election.cc:304] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:26.684197 597 raft_consensus.cc:2802] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:26.684355 597 raft_consensus.cc:695] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:26.684290 453 leader_election.cc:290] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:26.684638 597 consensus_queue.cc:237] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.685885 32314 leader_election.cc:304] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:26.686093 32314 raft_consensus.cc:2684] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:26.701156 340 ts_tablet_manager.cc:1905] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:26.702665 32128 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0773fc60d4a463596f536f15d58bf57 has no consensus state
I20250623 08:04:26.704732 32128 catalog_manager.cc:5582] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:26.706131 32129 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 4ea81642050141a9be84f151a5785abc
I20250623 08:04:26.706851 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_e851e10456a04a51814a556167f4777f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:26.707377 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e851e10456a04a51814a556167f4777f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:26.708599 32126 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2060a643d0da47efa868f52bdb331b1f consensus state has no leader
I20250623 08:04:26.708936 32126 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8b1e8627d7a64cc394552d28e372c826" } modify_external_catalogs: true
I20250623 08:04:26.709120 32126 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_8b1e8627d7a64cc394552d28e372c826" } modify_external_catalogs: true
I20250623 08:04:26.713510 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ea81642050141a9be84f151a5785abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.713811 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ea81642050141a9be84f151a5785abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.717645 340 ts_tablet_manager.cc:1918] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.717815 340 log.cc:1199] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/9c4dec1a48f84fb3bd7bbd757f7b0c1b
I20250623 08:04:26.718168 340 ts_tablet_manager.cc:1939] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.719019 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b (table default.Table_0b402e459582438db7f3ad072b276ee0 [id=562e3d3edc3f4341ae8909d9fcf2e6c2]) successfully deleted
I20250623 08:04:26.720710 340 tablet_replica.cc:331] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.720897 340 raft_consensus.cc:2241] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.721082 340 raft_consensus.cc:2270] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.722537 340 ts_tablet_manager.cc:1905] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.727985 366 raft_consensus.cc:491] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.728256 366 raft_consensus.cc:513] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.728374 32130 master_service.cc:948] ReplaceTablet: received request to replace tablet 2060a643d0da47efa868f52bdb331b1f from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:26.728973 366 leader_election.cc:290] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:26.731912 576 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: acquiring CA information for follower catalog manager: success
I20250623 08:04:26.736923 610 raft_consensus.cc:491] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.737201 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:26.737151 610 raft_consensus.cc:513] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.738626 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet ed5c81d9e56c4ee28546cedb9a579f31 (table default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]) successfully deleted
I20250623 08:04:26.738655 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04eda92e14144733ab71a577dd3bd2b0" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:26.739235 32314 leader_election.cc:343] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:26.739410 610 leader_election.cc:290] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:26.740743 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c4dec1a48f84fb3bd7bbd757f7b0c1b" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:26.740948 32655 raft_consensus.cc:2466] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
W20250623 08:04:26.741158 32589 leader_election.cc:343] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 35428260e05e447ebf88ea41e63cdd02
I20250623 08:04:26.741762 587 raft_consensus.cc:491] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:26.742947 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 (DEFAULT_TABLE table=default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.743572 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 32c82141edea4068b54755a85c6d7fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 162308da898d4dde8ab5b1ab01d0f393) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.743721 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a (DEFAULT_TABLE table=default.Table_63662f63088548e29e6db7281e66691e [id=ec1f9e35e20c47fcbe2d4b80c7abeabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.743813 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet 19536c16ff32460e9db571e4ce411bab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.743953 576 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250623 08:04:26.744087 32624 tablet_service.cc:1468] Processing CreateTablet for tablet 2ecf4af9e476495683bd63cac20d4672 (DEFAULT_TABLE table=default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.744306 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d with delete_type TABLET_DATA_DELETED (replaced by tablet 136e482995a24c5daec55a701f3091fa) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.745479 344 ts_tablet_manager.cc:1918] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:26.745640 344 log.cc:1199] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/04a3ae58d26142d58c4df4f52d7e3e2c
I20250623 08:04:26.746080 344 ts_tablet_manager.cc:1939] T 04a3ae58d26142d58c4df4f52d7e3e2c P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:26.747630 342 tablet_bootstrap.cc:492] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:26.747834 342 ts_tablet_manager.cc:1397] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.152s user 0.004s sys 0.004s
I20250623 08:04:26.748728 342 raft_consensus.cc:357] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.748914 342 raft_consensus.cc:383] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.748971 342 raft_consensus.cc:738] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.749217 342 consensus_queue.cc:260] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.749749 342 ts_tablet_manager.cc:1428] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:26.750087 342 tablet_bootstrap.cc:492] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:26.751892 342 tablet_bootstrap.cc:654] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.752002 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c1596e80225041099fb513ce477eae33" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:26.752185 32656 raft_consensus.cc:3058] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:26.752353 32122 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 175ee90133994154b7e4b58b0770db3b consensus state has no leader
I20250623 08:04:26.756119 32623 tablet_service.cc:1468] Processing CreateTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 (DEFAULT_TABLE table=default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.756367 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04eda92e14144733ab71a577dd3bd2b0" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:26.756953 32314 leader_election.cc:343] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:26.757093 32314 leader_election.cc:304] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:26.757573 32631 tablet_service.cc:1468] Processing CreateTablet for tablet f3d32798168345edb9898751d083057c (DEFAULT_TABLE table=default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:26.757782 32589 leader_election.cc:343] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 35428260e05e447ebf88ea41e63cdd02
I20250623 08:04:26.758149 610 raft_consensus.cc:2747] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:26.758131 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 38ecd4c243b840f4b66d8e01078dca64 (DEFAULT_TABLE table=default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.758304 32621 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ea81642050141a9be84f151a5785abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.758451 32625 tablet_service.cc:1515] Processing DeleteTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 with delete_type TABLET_DATA_DELETED (replaced by tablet 205da880685b49b3b8669f73dc2c563b) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.758553 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet ce1f3ef3690c4837b95b105733884f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.758658 32589 leader_election.cc:304] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:26.759074 339 tablet_bootstrap.cc:492] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.759068 366 raft_consensus.cc:2747] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:26.759202 339 ts_tablet_manager.cc:1397] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.147s user 0.011s sys 0.029s
I20250623 08:04:26.759922 339 raft_consensus.cc:357] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.760087 339 raft_consensus.cc:383] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.760154 339 raft_consensus.cc:738] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.760381 339 consensus_queue.cc:260] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.761662 587 raft_consensus.cc:513] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.762249 587 leader_election.cc:290] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:26.763273 339 ts_tablet_manager.cc:1428] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:26.763337 344 tablet_replica.cc:331] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:26.763509 344 raft_consensus.cc:2241] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.763684 344 raft_consensus.cc:2270] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.764179 344 ts_tablet_manager.cc:1905] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.765653 339 tablet_bootstrap.cc:492] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.766064 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 20e7a78cfc384033810603316b9e66bf (DEFAULT_TABLE table=default.Table_7987e1a82b644bd69724bc61cd84445a [id=96f6aedcc9704226b4b14e5548a35ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.766705 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (DEFAULT_TABLE table=default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.767216 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.767503 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet d67a8f1f570741ebb886c4d136ca40d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.767745 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet 35428260e05e447ebf88ea41e63cdd02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.768078 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet e90151394fb842498f4e780acab1f1aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.768597 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965f1f0a107b4653b6aac01e0d8e259a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.768689 339 tablet_bootstrap.cc:654] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.768882 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f9e0d1b85a94ae9ae86e53688a55db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.774875 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ecf4af9e476495683bd63cac20d4672. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.776309 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cb66c7a273a412db3ae77af9e30d036. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.777136 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3d32798168345edb9898751d083057c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.777617 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38ecd4c243b840f4b66d8e01078dca64. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:26.778470 32589 leader_election.cc:343] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: b6ad6cd629c4423ca997efb517cc975b
W20250623 08:04:26.778735 32589 leader_election.cc:343] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: b6ad6cd629c4423ca997efb517cc975b
I20250623 08:04:26.778847 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea702abb31fb41498ab61bbb05b1f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.780229 32589 leader_election.cc:304] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:26.781073 587 raft_consensus.cc:2747] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:26.781687 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 04a3ae58d26142d58c4df4f52d7e3e2c (table default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]) successfully deleted
I20250623 08:04:26.783020 32656 raft_consensus.cc:2466] T c1596e80225041099fb513ce477eae33 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:26.785214 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20e7a78cfc384033810603316b9e66bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.785451 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_b065e3e6335f41acbc48965305b45d13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:26.786033 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b065e3e6335f41acbc48965305b45d13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.799360 340 ts_tablet_manager.cc:1918] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.799618 340 log.cc:1199] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/0c77fd6e54904722acb2d1437c627ba8
I20250623 08:04:26.799969 340 ts_tablet_manager.cc:1939] T 0c77fd6e54904722acb2d1437c627ba8 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.801077 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 0c77fd6e54904722acb2d1437c627ba8 (table default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]) successfully deleted
I20250623 08:04:26.801440 340 tablet_replica.cc:331] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.801770 32123 master_service.cc:948] ReplaceTablet: received request to replace tablet 279624b8bb834b95a4f08045968ebfb4 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:26.803303 344 ts_tablet_manager.cc:1918] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.803472 344 log.cc:1199] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/69e772df590747ff841be5f218e55acf
I20250623 08:04:26.804194 340 raft_consensus.cc:2241] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.804378 340 raft_consensus.cc:2270] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.804940 340 ts_tablet_manager.cc:1905] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.811347 344 ts_tablet_manager.cc:1939] T 69e772df590747ff841be5f218e55acf P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:26.811937 344 tablet_replica.cc:331] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:26.812074 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 69e772df590747ff841be5f218e55acf (table default.Table_c3e871b94f3944d6b2b75626c3af94ec [id=a61a71f32ae444338b041f6e0a9fa286]) successfully deleted
I20250623 08:04:26.812152 344 raft_consensus.cc:2241] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:26.812445 344 pending_rounds.cc:70] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:26.812556 344 pending_rounds.cc:77] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727388935024640 op_type: NO_OP noop_request { }
I20250623 08:04:26.812703 344 raft_consensus.cc:2887] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:26.812844 344 raft_consensus.cc:2270] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.815985 344 ts_tablet_manager.cc:1905] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.822350 340 ts_tablet_manager.cc:1918] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.822507 340 log.cc:1199] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/a46c0a89fc014054a75ed198e824b5b6
I20250623 08:04:26.827566 340 ts_tablet_manager.cc:1939] T a46c0a89fc014054a75ed198e824b5b6 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.828691 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet a46c0a89fc014054a75ed198e824b5b6 (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]) successfully deleted
I20250623 08:04:26.830309 343 tablet_bootstrap.cc:654] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:26.831933 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 7da84969a6464b5caeaf47af0cbaaa4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da84969a6464b5caeaf47af0cbaaa4d already in progress: creating tablet
W20250623 08:04:26.832855 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 32c82141edea4068b54755a85c6d7fb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32c82141edea4068b54755a85c6d7fb8 already in progress: creating tablet
W20250623 08:04:26.833622 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 19536c16ff32460e9db571e4ce411bab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19536c16ff32460e9db571e4ce411bab already in progress: creating tablet
W20250623 08:04:26.834292 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 5629d11c44e241d598f3e519b2fe1a44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5629d11c44e241d598f3e519b2fe1a44 already in progress: creating tablet
W20250623 08:04:26.835105 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet ce1f3ef3690c4837b95b105733884f1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce1f3ef3690c4837b95b105733884f1f already in progress: creating tablet
I20250623 08:04:26.835897 341 tablet_replica.cc:331] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
W20250623 08:04:26.836364 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 4ea81642050141a9be84f151a5785abc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea81642050141a9be84f151a5785abc already in progress: creating tablet
I20250623 08:04:26.837157 341 raft_consensus.cc:2241] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.837381 341 raft_consensus.cc:2270] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.837960 341 ts_tablet_manager.cc:1905] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.838918 344 ts_tablet_manager.cc:1918] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:26.840533 344 log.cc:1199] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/5ca4957928fc4630aa8d8ca6110d7598
I20250623 08:04:26.840986 344 ts_tablet_manager.cc:1939] T 5ca4957928fc4630aa8d8ca6110d7598 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:26.843441 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 5ca4957928fc4630aa8d8ca6110d7598 (table default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]) successfully deleted
I20250623 08:04:26.843942 32122 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_cb9f889d49ac4ff8b2cfdbc50b28c547" } modify_external_catalogs: true
I20250623 08:04:26.844139 32122 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_cb9f889d49ac4ff8b2cfdbc50b28c547" } modify_external_catalogs: true
W20250623 08:04:26.844290 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 7da84969a6464b5caeaf47af0cbaaa4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da84969a6464b5caeaf47af0cbaaa4d already in progress: creating tablet
I20250623 08:04:26.845328 32121 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_24b73ca48b554c6698827e520b13b121" } modify_external_catalogs: true
I20250623 08:04:26.845501 32121 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_24b73ca48b554c6698827e520b13b121" } modify_external_catalogs: true
I20250623 08:04:26.847335 32124 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_71cce1d280bf41b7bae2f47aeec9a39e" } modify_external_catalogs: true
I20250623 08:04:26.847559 32124 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_71cce1d280bf41b7bae2f47aeec9a39e" } modify_external_catalogs: true
I20250623 08:04:26.848845 32126 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet ea702abb31fb41498ab61bbb05b1f7e2
I20250623 08:04:26.849558 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:26.850088 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.850982 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.851441 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.853664 32122 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 6667d3b14988433486f8f0b147062711
I20250623 08:04:26.854066 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_fe81102858f244af9a4a286eaf6e34bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.854233 32129 master_service.cc:948] ReplaceTablet: received request to replace tablet b980c746a3c042bfae90aa076acbb1df from {username='slave'} at 127.0.0.1:50426
W20250623 08:04:26.854595 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe81102858f244af9a4a286eaf6e34bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.855171 342 tablet_bootstrap.cc:492] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:26.855331 342 ts_tablet_manager.cc:1397] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.021s
I20250623 08:04:26.855813 343 tablet_bootstrap.cc:492] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.855976 343 ts_tablet_manager.cc:1397] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.313s user 0.004s sys 0.000s
I20250623 08:04:26.856132 342 raft_consensus.cc:357] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.856324 342 raft_consensus.cc:383] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.856419 342 raft_consensus.cc:738] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.856662 342 consensus_queue.cc:260] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:26.856709 343 raft_consensus.cc:357] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.856863 343 raft_consensus.cc:383] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.856963 343 raft_consensus.cc:738] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.857196 342 ts_tablet_manager.cc:1428] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:26.857210 343 consensus_queue.cc:260] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:26.857558 342 tablet_bootstrap.cc:492] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:26.859220 342 tablet_bootstrap.cc:654] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:26.860740 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet ea702abb31fb41498ab61bbb05b1f7e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea702abb31fb41498ab61bbb05b1f7e2 already in progress: creating tablet
I20250623 08:04:26.864279 32126 master_service.cc:948] ReplaceTablet: received request to replace tablet ae7c2ac546114a819a543f1667fb7844 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:26.864414 344 tablet_replica.cc:331] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: stopping tablet replica
W20250623 08:04:26.864456 32125 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d02f74c24a174b18b08a870a4e47fefb consensus state has no leader
I20250623 08:04:26.864634 344 raft_consensus.cc:2241] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.864776 344 raft_consensus.cc:2270] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.864802 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_d226f9dab24f45ae896579983b3d1a67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:26.865278 344 ts_tablet_manager.cc:1905] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:26.865296 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d226f9dab24f45ae896579983b3d1a67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:26.866549 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 32c82141edea4068b54755a85c6d7fb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32c82141edea4068b54755a85c6d7fb8 already in progress: creating tablet
W20250623 08:04:26.866798 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 19536c16ff32460e9db571e4ce411bab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19536c16ff32460e9db571e4ce411bab already in progress: creating tablet
W20250623 08:04:26.867002 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 5629d11c44e241d598f3e519b2fe1a44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5629d11c44e241d598f3e519b2fe1a44 already in progress: creating tablet
W20250623 08:04:26.867229 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet ce1f3ef3690c4837b95b105733884f1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce1f3ef3690c4837b95b105733884f1f already in progress: creating tablet
I20250623 08:04:26.868038 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 with delete_type TABLET_DATA_DELETED (replaced by tablet d02f74c24a174b18b08a870a4e47fefb) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.866921 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:26.868911 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.857751 343 ts_tablet_manager.cc:1428] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:26.869782 343 tablet_bootstrap.cc:492] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.860172 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.866730 32624 tablet_service.cc:1515] Processing DeleteTablet for tablet a46c0a89fc014054a75ed198e824b5b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 228375592aee48a1aa21e1d16962b663) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.873339 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a22e12b2bd5b44aab29760196b28bd06" } new_table_name: "default.Table_a6ef1bf66cd24835896dabd3af080a6a" modify_external_catalogs: true
I20250623 08:04:26.873478 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 with delete_type TABLET_DATA_DELETED (replaced by tablet d02f74c24a174b18b08a870a4e47fefb) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.873749 340 tablet_replica.cc:331] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.873921 340 raft_consensus.cc:2241] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.874099 340 raft_consensus.cc:2270] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.874852 340 ts_tablet_manager.cc:1905] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.875653 342 tablet_bootstrap.cc:492] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:26.875809 342 ts_tablet_manager.cc:1397] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20250623 08:04:26.876518 342 raft_consensus.cc:357] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.876709 342 raft_consensus.cc:383] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.876777 342 raft_consensus.cc:738] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.877004 342 consensus_queue.cc:260] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.877485 342 ts_tablet_manager.cc:1428] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:26.877591 343 tablet_bootstrap.cc:654] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.877775 342 tablet_bootstrap.cc:492] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:26.879386 342 tablet_bootstrap.cc:654] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.883363 32488 tablet_service.cc:1468] Processing CreateTablet for tablet c0773fc60d4a463596f536f15d58bf57 (DEFAULT_TABLE table=default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.883726 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0773fc60d4a463596f536f15d58bf57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.885561 340 ts_tablet_manager.cc:1918] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.885739 340 log.cc:1199] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/7fea0b5aa0c1403fb3ef3ab9ab040035
I20250623 08:04:26.886484 340 ts_tablet_manager.cc:1939] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:26.890852 32350 tablet_service.cc:1468] Processing CreateTablet for tablet c0773fc60d4a463596f536f15d58bf57 (DEFAULT_TABLE table=default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.891144 32489 tablet_service.cc:1468] Processing CreateTablet for tablet cdec6e9573b041e0a57d1e0e0239d97f (DEFAULT_TABLE table=default.Table_48cbdc58fd034e6390087ed7c6346d24 [id=ef1a8122b8044388b4ae24a03da26837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.891245 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0773fc60d4a463596f536f15d58bf57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.891531 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdec6e9573b041e0a57d1e0e0239d97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.891780 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 48da1e4759bb4fea87a45ad8a428b6e7 (DEFAULT_TABLE table=default.Table_c382db2a0c144ec4abd51b3e927e0a17 [id=461d38c05dc24245b0c59cb73b42a394]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.892174 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48da1e4759bb4fea87a45ad8a428b6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.895081 32624 tablet_service.cc:1468] Processing CreateTablet for tablet c0773fc60d4a463596f536f15d58bf57 (DEFAULT_TABLE table=default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.895435 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0773fc60d4a463596f536f15d58bf57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.898083 341 ts_tablet_manager.cc:1918] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.898217 341 log.cc:1199] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/125784661a7b4e4ba6a710b950e004f8
I20250623 08:04:26.898574 341 ts_tablet_manager.cc:1939] T 125784661a7b4e4ba6a710b950e004f8 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:26.899127 32623 tablet_service.cc:1468] Processing CreateTablet for tablet cdec6e9573b041e0a57d1e0e0239d97f (DEFAULT_TABLE table=default.Table_48cbdc58fd034e6390087ed7c6346d24 [id=ef1a8122b8044388b4ae24a03da26837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.899458 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 48da1e4759bb4fea87a45ad8a428b6e7 (DEFAULT_TABLE table=default.Table_c382db2a0c144ec4abd51b3e927e0a17 [id=461d38c05dc24245b0c59cb73b42a394]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.899597 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdec6e9573b041e0a57d1e0e0239d97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.899830 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48da1e4759bb4fea87a45ad8a428b6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.900758 32354 tablet_service.cc:1468] Processing CreateTablet for tablet cdec6e9573b041e0a57d1e0e0239d97f (DEFAULT_TABLE table=default.Table_48cbdc58fd034e6390087ed7c6346d24 [id=ef1a8122b8044388b4ae24a03da26837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.901041 32495 tablet_service.cc:1468] Processing CreateTablet for tablet 712abfca1fa4489f882534c33b9db1be (DEFAULT_TABLE table=default.Table_bc69503004f74bf59fd961b4f209baa6 [id=836b8d720fae4622a556600a1a9c1d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.901129 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdec6e9573b041e0a57d1e0e0239d97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.901304 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]) successfully deleted
I20250623 08:04:26.901368 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712abfca1fa4489f882534c33b9db1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.901728 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 7adec1482119405daca4aa603770ae22 (DEFAULT_TABLE table=default.Table_a615b39019e1478aa69bd46fc2e4a4a9 [id=1e194b3160b64acabac38acfd50a64b3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:26.901995 32117 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_7e39ecbac4a8433e8ae98b0db22c30ea [id=8b20894a91c54a3eb08d5e1b4a42d269], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8c01121f2e644d6aa6e8ec228bf75be9 consensus state has no leader
I20250623 08:04:26.902109 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7adec1482119405daca4aa603770ae22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.903071 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 48da1e4759bb4fea87a45ad8a428b6e7 (DEFAULT_TABLE table=default.Table_c382db2a0c144ec4abd51b3e927e0a17 [id=461d38c05dc24245b0c59cb73b42a394]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.903396 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48da1e4759bb4fea87a45ad8a428b6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.906996 32497 tablet_service.cc:1468] Processing CreateTablet for tablet c63029399994480ba7ccabcadc1ee5e0 (DEFAULT_TABLE table=default.Table_a22e12b2bd5b44aab29760196b28bd06 [id=33768fa88bfc4cfba1359030af5e77c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.907017 32348 tablet_service.cc:1468] Processing CreateTablet for tablet c63029399994480ba7ccabcadc1ee5e0 (DEFAULT_TABLE table=default.Table_a22e12b2bd5b44aab29760196b28bd06 [id=33768fa88bfc4cfba1359030af5e77c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.907316 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c63029399994480ba7ccabcadc1ee5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.907357 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c63029399994480ba7ccabcadc1ee5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.908166 32622 tablet_service.cc:1468] Processing CreateTablet for tablet c63029399994480ba7ccabcadc1ee5e0 (DEFAULT_TABLE table=default.Table_a22e12b2bd5b44aab29760196b28bd06 [id=33768fa88bfc4cfba1359030af5e77c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.908533 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c63029399994480ba7ccabcadc1ee5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.910836 32356 tablet_service.cc:1468] Processing CreateTablet for tablet 7adec1482119405daca4aa603770ae22 (DEFAULT_TABLE table=default.Table_a615b39019e1478aa69bd46fc2e4a4a9 [id=1e194b3160b64acabac38acfd50a64b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.911017 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 7adec1482119405daca4aa603770ae22 (DEFAULT_TABLE table=default.Table_a615b39019e1478aa69bd46fc2e4a4a9 [id=1e194b3160b64acabac38acfd50a64b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.911198 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7adec1482119405daca4aa603770ae22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.911396 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7adec1482119405daca4aa603770ae22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.917071 339 tablet_bootstrap.cc:492] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:26.917285 339 ts_tablet_manager.cc:1397] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.152s user 0.010s sys 0.008s
I20250623 08:04:26.918270 339 raft_consensus.cc:357] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.918534 339 raft_consensus.cc:383] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.918633 339 raft_consensus.cc:738] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.922271 32488 tablet_service.cc:1468] Processing CreateTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 (DEFAULT_TABLE table=default.Table_d44a2132435f43e7847bfa349645a605 [id=b42459e9ca5e475cb5083d68bb6349be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.922660 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79731106915d4cebb9dcc3c10386ffa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.922596 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 (DEFAULT_TABLE table=default.Table_d44a2132435f43e7847bfa349645a605 [id=b42459e9ca5e475cb5083d68bb6349be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.922955 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79731106915d4cebb9dcc3c10386ffa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.924484 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 712abfca1fa4489f882534c33b9db1be (DEFAULT_TABLE table=default.Table_bc69503004f74bf59fd961b4f209baa6 [id=836b8d720fae4622a556600a1a9c1d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.924863 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712abfca1fa4489f882534c33b9db1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.927079 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 (DEFAULT_TABLE table=default.Table_d44a2132435f43e7847bfa349645a605 [id=b42459e9ca5e475cb5083d68bb6349be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.929840 32495 tablet_service.cc:1468] Processing CreateTablet for tablet 31ecc74317d449e5b557b07a4fd14895 (DEFAULT_TABLE table=default.Table_7f123c9cf6f54d22ae18a9d6ce7d9d9c [id=6e61ff2d881e41dcbf03374e0f48b4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.930192 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31ecc74317d449e5b557b07a4fd14895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.937252 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79731106915d4cebb9dcc3c10386ffa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.937346 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 31ecc74317d449e5b557b07a4fd14895 (DEFAULT_TABLE table=default.Table_7f123c9cf6f54d22ae18a9d6ce7d9d9c [id=6e61ff2d881e41dcbf03374e0f48b4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.937701 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31ecc74317d449e5b557b07a4fd14895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.940577 339 consensus_queue.cc:260] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.941164 339 ts_tablet_manager.cc:1428] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20250623 08:04:26.943301 343 tablet_bootstrap.cc:492] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:26.943477 343 ts_tablet_manager.cc:1397] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20250623 08:04:26.944224 343 raft_consensus.cc:357] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.944445 343 raft_consensus.cc:383] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:26.944517 343 raft_consensus.cc:738] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:26.944770 343 consensus_queue.cc:260] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:26.945395 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 31ecc74317d449e5b557b07a4fd14895 (DEFAULT_TABLE table=default.Table_7f123c9cf6f54d22ae18a9d6ce7d9d9c [id=6e61ff2d881e41dcbf03374e0f48b4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.945770 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31ecc74317d449e5b557b07a4fd14895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.946193 343 ts_tablet_manager.cc:1428] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:26.946483 343 tablet_bootstrap.cc:492] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:26.948738 343 tablet_bootstrap.cc:654] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:26.954211 339 tablet_bootstrap.cc:492] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:26.957180 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 712abfca1fa4489f882534c33b9db1be (DEFAULT_TABLE table=default.Table_bc69503004f74bf59fd961b4f209baa6 [id=836b8d720fae4622a556600a1a9c1d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:26.958072 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712abfca1fa4489f882534c33b9db1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:26.969237 32125 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_63662f63088548e29e6db7281e66691e" } new_table_name: "default.Table_5f159a56c60144b99a6847ec6080623e" modify_external_catalogs: true
I20250623 08:04:26.970896 32128 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d44a2132435f43e7847bfa349645a605" } modify_external_catalogs: true
I20250623 08:04:26.971170 32128 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d44a2132435f43e7847bfa349645a605" } modify_external_catalogs: true
W20250623 08:04:26.972687 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c63029399994480ba7ccabcadc1ee5e0 consensus state has no leader
I20250623 08:04:26.973004 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_b065e3e6335f41acbc48965305b45d13" } new_table_name: "default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f" modify_external_catalogs: true
I20250623 08:04:26.974160 32130 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 2060a643d0da47efa868f52bdb331b1f
I20250623 08:04:26.975207 32130 catalog_manager.cc:6652] ReplaceTablet: tablet 2060a643d0da47efa868f52bdb331b1f deleted and replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1
I20250623 08:04:26.977547 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_79f69fca0b25496e896d93fdd78e14b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:26.978085 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79f69fca0b25496e896d93fdd78e14b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:26.988395 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 151) took 1233 ms. Trace:
I20250623 08:04:26.988618 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet 2060a643d0da47efa868f52bdb331b1f with delete_type TABLET_DATA_DELETED (replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:26.988579 341 rpcz_store.cc:276] 0623 08:04:25.754456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:25.759734 (+ 5278us) service_pool.cc:225] Handling call
0623 08:04:26.734268 (+974534us) ts_tablet_manager.cc:1145] Deleting tablet 125784661a7b4e4ba6a710b950e004f8
0623 08:04:26.746690 (+ 12422us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:26.875042 (+128352us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:26.897974 (+ 22932us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:26.988320 (+ 90346us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":969,"tablet-delete.queue_time_us":974233}
I20250623 08:04:26.990506 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 125784661a7b4e4ba6a710b950e004f8 (table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]) successfully deleted
I20250623 08:04:26.990816 32347 tablet_service.cc:1515] Processing DeleteTablet for tablet 04eda92e14144733ab71a577dd3bd2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.991103 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.991324 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 2060a643d0da47efa868f52bdb331b1f with delete_type TABLET_DATA_DELETED (replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:26.991576 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet 2060a643d0da47efa868f52bdb331b1f with delete_type TABLET_DATA_DELETED (replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:26.991822 340 tablet_replica.cc:331] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:26.992030 340 raft_consensus.cc:2241] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:26.992206 340 raft_consensus.cc:2270] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:26.993659 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 156) took 1199 ms. Trace:
I20250623 08:04:26.994020 340 ts_tablet_manager.cc:1905] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:26.995888 32123 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 279624b8bb834b95a4f08045968ebfb4
I20250623 08:04:26.996301 32123 catalog_manager.cc:6652] ReplaceTablet: tablet 279624b8bb834b95a4f08045968ebfb4 deleted and replaced by tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f
I20250623 08:04:26.999691 344 ts_tablet_manager.cc:1918] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:26.999840 344 log.cc:1199] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/2eb99713f9214e6488d5c6e140ac263c
I20250623 08:04:26.999848 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_dbd34051348e4440be23ffa0df7b19cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:27.000136 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet 279624b8bb834b95a4f08045968ebfb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f) from {username='slave'} at 127.0.0.1:55518
W20250623 08:04:27.000418 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbd34051348e4440be23ffa0df7b19cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.001483 341 rpcz_store.cc:276] 0623 08:04:25.794104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:25.794156 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:26.993470 (+1199314us) ts_tablet_manager.cc:1145] Deleting tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa
0623 08:04:26.993482 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:26.993610 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199049}
I20250623 08:04:27.002002 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 157) took 1198 ms. Trace:
I20250623 08:04:27.002116 341 rpcz_store.cc:276] 0623 08:04:25.803594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:25.803656 (+ 62us) service_pool.cc:225] Handling call
0623 08:04:27.001804 (+1198148us) ts_tablet_manager.cc:1145] Deleting tablet 67c44697ecab4dd9b43f544a602897ca
0623 08:04:27.001815 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:27.001930 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197902}
I20250623 08:04:27.002468 341 tablet_replica.cc:331] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.002641 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet c7385d86316f4a3ab68a77d18080ec20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.004030 342 tablet_bootstrap.cc:492] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.004173 342 ts_tablet_manager.cc:1397] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.126s user 0.014s sys 0.018s
I20250623 08:04:27.005015 342 raft_consensus.cc:357] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.005160 32126 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet ae7c2ac546114a819a543f1667fb7844
I20250623 08:04:27.005196 342 raft_consensus.cc:383] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.005302 342 raft_consensus.cc:738] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.005554 342 consensus_queue.cc:260] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.005712 32126 catalog_manager.cc:6652] ReplaceTablet: tablet ae7c2ac546114a819a543f1667fb7844 deleted and replaced by tablet d98654c29b9c4e75af6a213cafd4c2c9
I20250623 08:04:27.006065 342 ts_tablet_manager.cc:1428] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:27.006033 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_a4ca66fa08cc441ab5571f9968f30914"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:27.006434 342 tablet_bootstrap.cc:492] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:27.006610 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4ca66fa08cc441ab5571f9968f30914 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.007786 32121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 417a050eada4403ebbddff255fb6fb5a
I20250623 08:04:27.008096 342 tablet_bootstrap.cc:654] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.008330 32121 master_service.cc:948] ReplaceTablet: received request to replace tablet 175ee90133994154b7e4b58b0770db3b from {username='slave'} at 127.0.0.1:50426
W20250623 08:04:27.013765 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa already in progress: creating tablet
I20250623 08:04:27.014137 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d with delete_type TABLET_DATA_DELETED (replaced by tablet c7fcf583064f4538913d9157e93068dd) from {username='slave'} at 127.0.0.1:35526
W20250623 08:04:27.014155 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 67c44697ecab4dd9b43f544a602897ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67c44697ecab4dd9b43f544a602897ca already in progress: creating tablet
I20250623 08:04:27.014613 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet ae7c2ac546114a819a543f1667fb7844 with delete_type TABLET_DATA_DELETED (replaced by tablet d98654c29b9c4e75af6a213cafd4c2c9) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.000228 344 ts_tablet_manager.cc:1939] T 2eb99713f9214e6488d5c6e140ac263c P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:27.014870 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 279624b8bb834b95a4f08045968ebfb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.015149 343 tablet_bootstrap.cc:492] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.015295 343 ts_tablet_manager.cc:1397] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.008s
I20250623 08:04:27.016062 339 tablet_bootstrap.cc:654] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.016096 343 raft_consensus.cc:357] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.016263 343 raft_consensus.cc:383] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.016364 343 raft_consensus.cc:738] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.016801 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet ae7c2ac546114a819a543f1667fb7844 with delete_type TABLET_DATA_DELETED (replaced by tablet d98654c29b9c4e75af6a213cafd4c2c9) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.016845 343 consensus_queue.cc:260] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.017419 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet ae7c2ac546114a819a543f1667fb7844 with delete_type TABLET_DATA_DELETED (replaced by tablet d98654c29b9c4e75af6a213cafd4c2c9) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.017947 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet 417a050eada4403ebbddff255fb6fb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.019505 343 ts_tablet_manager.cc:1428] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:27.019835 343 tablet_bootstrap.cc:492] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.002686 341 raft_consensus.cc:2241] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.001012 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 279624b8bb834b95a4f08045968ebfb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.021399 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet 417a050eada4403ebbddff255fb6fb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.021844 341 raft_consensus.cc:2270] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.022128 344 tablet_replica.cc:331] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:27.021595 343 tablet_bootstrap.cc:654] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.023062 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 417a050eada4403ebbddff255fb6fb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.023535 341 ts_tablet_manager.cc:1905] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.026515 344 raft_consensus.cc:2241] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.026763 344 raft_consensus.cc:2270] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.027392 344 ts_tablet_manager.cc:1905] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.029136 32124 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet b980c746a3c042bfae90aa076acbb1df
I20250623 08:04:27.030185 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet b980c746a3c042bfae90aa076acbb1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.030344 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet b980c746a3c042bfae90aa076acbb1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.031323 32124 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_9788bd7b3d2848368b935832a2cffaed" } new_table_name: "default.Table_c30b48097bd643b18b2a744839780b9e" modify_external_catalogs: true
I20250623 08:04:27.032850 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_c2afde03c007450390a0466fe1d39344"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.033396 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2afde03c007450390a0466fe1d39344 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.034256 339 tablet_bootstrap.cc:492] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:27.034427 339 ts_tablet_manager.cc:1397] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.000s
I20250623 08:04:27.035692 339 raft_consensus.cc:357] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.035926 339 raft_consensus.cc:383] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.036026 339 raft_consensus.cc:738] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.036312 339 consensus_queue.cc:260] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.037518 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet b980c746a3c042bfae90aa076acbb1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.038628 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 2eb99713f9214e6488d5c6e140ac263c (table default.Table_2807052f51c24614b4068e8113db2abd [id=9e90ae34db704c408759a60271e0bf82]) successfully deleted
I20250623 08:04:27.039268 339 ts_tablet_manager.cc:1428] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:27.039425 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 04a3ae58d26142d58c4df4f52d7e3e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.041841 339 tablet_bootstrap.cc:492] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:27.045744 32125 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 965f1f0a107b4653b6aac01e0d8e259a consensus state has no leader
I20250623 08:04:27.045754 340 ts_tablet_manager.cc:1918] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.045881 340 log.cc:1199] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/04eda92e14144733ab71a577dd3bd2b0
I20250623 08:04:27.046197 340 ts_tablet_manager.cc:1939] T 04eda92e14144733ab71a577dd3bd2b0 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:27.046116 32125 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_640be163e2114433bd7079aa3a602116"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.046698 32125 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_640be163e2114433bd7079aa3a602116 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.047820 342 tablet_bootstrap.cc:492] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.047977 342 ts_tablet_manager.cc:1397] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.007s
I20250623 08:04:27.048345 339 tablet_bootstrap.cc:654] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:27.048592 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1c82e4d2db64026a04c5c84bc1e09ec has no consensus state
I20250623 08:04:27.048755 610 raft_consensus.cc:491] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.048790 342 raft_consensus.cc:357] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.048903 610 raft_consensus.cc:513] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:27.048987 342 raft_consensus.cc:383] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.049050 342 raft_consensus.cc:738] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.049297 342 consensus_queue.cc:260] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.049372 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 79731106915d4cebb9dcc3c10386ffa9
I20250623 08:04:27.049533 610 leader_election.cc:290] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:27.049808 342 ts_tablet_manager.cc:1428] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:27.050707 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db9480bb07343fe935fe9bfc98bca77" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:27.051237 32314 leader_election.cc:343] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.052139 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.052376 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.052378 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db9480bb07343fe935fe9bfc98bca77" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:27.052542 32520 raft_consensus.cc:2466] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:27.053020 342 tablet_bootstrap.cc:492] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.054015 32314 leader_election.cc:304] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:27.054502 610 raft_consensus.cc:2802] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:27.054592 610 raft_consensus.cc:491] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:27.054693 610 raft_consensus.cc:3058] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.055840 342 tablet_bootstrap.cc:654] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.056761 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.058208 610 raft_consensus.cc:513] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:27.059505 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db9480bb07343fe935fe9bfc98bca77" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:27.059756 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db9480bb07343fe935fe9bfc98bca77" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:27.059914 32520 raft_consensus.cc:3058] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.060760 32128 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_dfc3da325f6849bfba88293a5eabb3e4" } modify_external_catalogs: true
I20250623 08:04:27.060976 32128 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_dfc3da325f6849bfba88293a5eabb3e4" } modify_external_catalogs: true
I20250623 08:04:27.061604 32122 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_347f265adebd45bdb27956d1affc3e89" } modify_external_catalogs: true
I20250623 08:04:27.061594 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_e7f8192c00524b5899152fa3f717b898"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:27.061780 32122 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_347f265adebd45bdb27956d1affc3e89" } modify_external_catalogs: true
W20250623 08:04:27.062209 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7f8192c00524b5899152fa3f717b898 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:27.063213 32314 leader_election.cc:343] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.063861 32520 raft_consensus.cc:2466] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:27.064268 32314 leader_election.cc:304] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:27.064711 610 leader_election.cc:290] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:27.064922 610 raft_consensus.cc:2802] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:27.065182 610 raft_consensus.cc:695] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:27.065649 610 consensus_queue.cc:237] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:27.065812 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_0bb2584ec3e0467194a9591bbe5ae477"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.066448 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bb2584ec3e0467194a9591bbe5ae477 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.077337 343 tablet_bootstrap.cc:492] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.077610 343 ts_tablet_manager.cc:1397] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.003s
I20250623 08:04:27.078516 343 raft_consensus.cc:357] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.078817 343 raft_consensus.cc:383] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.078949 343 raft_consensus.cc:738] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.079267 343 consensus_queue.cc:260] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:27.082533 32124 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 162308da898d4dde8ab5b1ab01d0f393 has no consensus state
I20250623 08:04:27.082576 343 ts_tablet_manager.cc:1428] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:27.084357 32124 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_5174877154cc448c98758389a337e263" } new_table_name: "default.Table_1428aeeb452648f0beaadc1b2f78c83b" modify_external_catalogs: true
I20250623 08:04:27.085304 343 tablet_bootstrap.cc:492] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.091655 339 tablet_bootstrap.cc:492] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:27.091832 339 ts_tablet_manager.cc:1397] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.001s
I20250623 08:04:27.092648 339 raft_consensus.cc:357] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.092849 339 raft_consensus.cc:383] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.092918 339 raft_consensus.cc:738] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.093137 339 consensus_queue.cc:260] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.099326 587 raft_consensus.cc:491] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.099556 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 2eb99713f9214e6488d5c6e140ac263c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.099503 587 raft_consensus.cc:513] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.100106 587 leader_election.cc:290] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:27.101308 32589 leader_election.cc:343] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: e90151394fb842498f4e780acab1f1aa
W20250623 08:04:27.101528 32589 leader_election.cc:343] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: e90151394fb842498f4e780acab1f1aa
I20250623 08:04:27.101579 340 tablet_replica.cc:331] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:27.101636 32589 leader_election.cc:304] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.101774 340 raft_consensus.cc:2241] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.101922 340 raft_consensus.cc:2270] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.102571 587 raft_consensus.cc:2747] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.102735 340 ts_tablet_manager.cc:1905] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.106561 341 ts_tablet_manager.cc:1918] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.107532 344 ts_tablet_manager.cc:1918] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.107559 32350 tablet_service.cc:1468] Processing CreateTablet for tablet a1c82e4d2db64026a04c5c84bc1e09ec (DEFAULT_TABLE table=default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.107780 341 log.cc:1199] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/5ca4957928fc4630aa8d8ca6110d7598
I20250623 08:04:27.107964 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1c82e4d2db64026a04c5c84bc1e09ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.108354 341 ts_tablet_manager.cc:1939] T 5ca4957928fc4630aa8d8ca6110d7598 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.107649 32495 tablet_service.cc:1468] Processing CreateTablet for tablet a1c82e4d2db64026a04c5c84bc1e09ec (DEFAULT_TABLE table=default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.108780 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1c82e4d2db64026a04c5c84bc1e09ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.109294 32488 tablet_service.cc:1468] Processing CreateTablet for tablet a9ea74d20c5f4eefbcb8508ed1774026 (DEFAULT_TABLE table=default.Table_e851e10456a04a51814a556167f4777f [id=ced0b4cf62924dbfadad02109349bea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.109634 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ea74d20c5f4eefbcb8508ed1774026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.112609 32358 tablet_service.cc:1468] Processing CreateTablet for tablet a9ea74d20c5f4eefbcb8508ed1774026 (DEFAULT_TABLE table=default.Table_e851e10456a04a51814a556167f4777f [id=ced0b4cf62924dbfadad02109349bea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.113039 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ea74d20c5f4eefbcb8508ed1774026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.116246 344 log.cc:1199] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9c4dec1a48f84fb3bd7bbd757f7b0c1b
I20250623 08:04:27.116664 32623 tablet_service.cc:1468] Processing CreateTablet for tablet a9ea74d20c5f4eefbcb8508ed1774026 (DEFAULT_TABLE table=default.Table_e851e10456a04a51814a556167f4777f [id=ced0b4cf62924dbfadad02109349bea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.116789 342 tablet_bootstrap.cc:492] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.117027 342 ts_tablet_manager.cc:1397] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.010s
I20250623 08:04:27.117048 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ea74d20c5f4eefbcb8508ed1774026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.117264 344 ts_tablet_manager.cc:1939] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:27.117843 342 raft_consensus.cc:357] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.118021 342 raft_consensus.cc:383] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.118100 342 raft_consensus.cc:738] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.118330 342 consensus_queue.cc:260] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.118476 341 tablet_replica.cc:331] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.118685 341 raft_consensus.cc:2241] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.118918 342 ts_tablet_manager.cc:1428] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:27.108424 32631 tablet_service.cc:1468] Processing CreateTablet for tablet a1c82e4d2db64026a04c5c84bc1e09ec (DEFAULT_TABLE table=default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.119197 342 tablet_bootstrap.cc:492] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.119556 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1c82e4d2db64026a04c5c84bc1e09ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.120208 32488 tablet_service.cc:1468] Processing CreateTablet for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (DEFAULT_TABLE table=default.Table_fe75261182e243ebbbcfbf6e636ba134 [id=5d4bcecfd3904412b0aa6fab7c75d584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.120918 342 tablet_bootstrap.cc:654] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.121848 32354 tablet_service.cc:1468] Processing CreateTablet for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (DEFAULT_TABLE table=default.Table_fe75261182e243ebbbcfbf6e636ba134 [id=5d4bcecfd3904412b0aa6fab7c75d584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.122113 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 2f62ec83c88e43438b963e740d2c7004 (DEFAULT_TABLE table=default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac [id=65c4e79e095b41b2b2a42e9858485423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.122334 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdc9d47f2232403fa71e14a6fe6f33c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.123203 32489 tablet_service.cc:1468] Processing CreateTablet for tablet f4a0b9402a264800a1ea39707c10da0c (DEFAULT_TABLE table=default.Table_244bcdc8d69b4f61a337dac403ffe6cb [id=b3ab55b66d754642aed2f746d9a35807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.123359 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 (DEFAULT_TABLE table=default.Table_3ef2e86e99164cce8e882e6954e2ec17 [id=dbbd669a72954d1aaf169b5870a85fc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.123898 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 8dc696236a15420e9666cca708068fb2 (DEFAULT_TABLE table=default.Table_f81a1165815543b781251c1ecfd06779 [id=491b95490f8c45eda181ea0cafd42368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.124250 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 162308da898d4dde8ab5b1ab01d0f393 (DEFAULT_TABLE table=default.Table_c30b48097bd643b18b2a744839780b9e [id=7e59ad2235c1418d97f2b73ff2a48300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.124646 32491 tablet_service.cc:1468] Processing CreateTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 (DEFAULT_TABLE table=default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.124853 32636 tablet_service.cc:1468] Processing CreateTablet for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (DEFAULT_TABLE table=default.Table_fe75261182e243ebbbcfbf6e636ba134 [id=5d4bcecfd3904412b0aa6fab7c75d584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.125077 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 205da880685b49b3b8669f73dc2c563b (DEFAULT_TABLE table=default.Table_4d4535cab44545839eb3a559385e395a [id=bb849e86d7084959830c68207fb99061]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.125203 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdc9d47f2232403fa71e14a6fe6f33c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.125491 32496 tablet_service.cc:1468] Processing CreateTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 (DEFAULT_TABLE table=default.Table_4550e74cc6e845719c1466d3acd1b723 [id=587d6896c7dd40838ce8ad01026379a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.125557 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 04eda92e14144733ab71a577dd3bd2b0 (table default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a [id=89a3fc1cb42f4902a69ea52945cedc0c]) successfully deleted
I20250623 08:04:27.125828 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 5ca4957928fc4630aa8d8ca6110d7598 (table default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]) successfully deleted
I20250623 08:04:27.126266 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 025365d2fa0a43dd983d395a05cc2af1 (DEFAULT_TABLE table=default.Table_2c99afea2b6e4963985dab32b545779e [id=e6cd13b44efa4cbbbdd2c14d8ea22267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.126348 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b (table default.Table_0b402e459582438db7f3ad072b276ee0 [id=562e3d3edc3f4341ae8909d9fcf2e6c2]) successfully deleted
I20250623 08:04:27.127288 344 tablet_replica.cc:331] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:27.127518 344 raft_consensus.cc:2241] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.127688 344 raft_consensus.cc:2270] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.128225 344 ts_tablet_manager.cc:1905] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.130180 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 025365d2fa0a43dd983d395a05cc2af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.131101 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 2f62ec83c88e43438b963e740d2c7004 (DEFAULT_TABLE table=default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac [id=65c4e79e095b41b2b2a42e9858485423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.131453 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f62ec83c88e43438b963e740d2c7004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.132342 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.133765 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dc696236a15420e9666cca708068fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.134232 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 2f62ec83c88e43438b963e740d2c7004 (DEFAULT_TABLE table=default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac [id=65c4e79e095b41b2b2a42e9858485423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.134567 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f62ec83c88e43438b963e740d2c7004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.137918 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162308da898d4dde8ab5b1ab01d0f393. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.139906 32348 tablet_service.cc:1468] Processing CreateTablet for tablet f4a0b9402a264800a1ea39707c10da0c (DEFAULT_TABLE table=default.Table_244bcdc8d69b4f61a337dac403ffe6cb [id=b3ab55b66d754642aed2f746d9a35807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.141036 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a0b9402a264800a1ea39707c10da0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.143066 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205da880685b49b3b8669f73dc2c563b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.122125 32495 tablet_service.cc:1468] Processing CreateTablet for tablet 025365d2fa0a43dd983d395a05cc2af1 (DEFAULT_TABLE table=default.Table_2c99afea2b6e4963985dab32b545779e [id=e6cd13b44efa4cbbbdd2c14d8ea22267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.143632 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 025365d2fa0a43dd983d395a05cc2af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.144389 32356 tablet_service.cc:1468] Processing CreateTablet for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 (DEFAULT_TABLE table=default.Table_3ef2e86e99164cce8e882e6954e2ec17 [id=dbbd669a72954d1aaf169b5870a85fc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.145130 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.146967 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c8d480b8ba49a1be140b0e0a79f4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.150245 32622 tablet_service.cc:1468] Processing CreateTablet for tablet f4a0b9402a264800a1ea39707c10da0c (DEFAULT_TABLE table=default.Table_244bcdc8d69b4f61a337dac403ffe6cb [id=b3ab55b66d754642aed2f746d9a35807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.150652 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a0b9402a264800a1ea39707c10da0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.151192 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 162308da898d4dde8ab5b1ab01d0f393 (DEFAULT_TABLE table=default.Table_c30b48097bd643b18b2a744839780b9e [id=7e59ad2235c1418d97f2b73ff2a48300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.151806 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 8dc696236a15420e9666cca708068fb2 (DEFAULT_TABLE table=default.Table_f81a1165815543b781251c1ecfd06779 [id=491b95490f8c45eda181ea0cafd42368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.152253 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162308da898d4dde8ab5b1ab01d0f393. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.153509 32357 tablet_service.cc:1468] Processing CreateTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 (DEFAULT_TABLE table=default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.153851 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 (DEFAULT_TABLE table=default.Table_3ef2e86e99164cce8e882e6954e2ec17 [id=dbbd669a72954d1aaf169b5870a85fc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.154188 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dc696236a15420e9666cca708068fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.154969 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ed4cb252b1452087f8ee53ec78d985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.155001 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.156949 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 205da880685b49b3b8669f73dc2c563b (DEFAULT_TABLE table=default.Table_4d4535cab44545839eb3a559385e395a [id=bb849e86d7084959830c68207fb99061]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.157262 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205da880685b49b3b8669f73dc2c563b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.158221 32359 tablet_service.cc:1468] Processing CreateTablet for tablet f279b63b55c64e548ded27b02025c9f5 (DEFAULT_TABLE table=default.Table_6e40e423312e4f9e95174f277b8897ce [id=7038d3807ece414f8a8e76bfe107c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.158577 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f279b63b55c64e548ded27b02025c9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.158515 32624 tablet_service.cc:1468] Processing CreateTablet for tablet 025365d2fa0a43dd983d395a05cc2af1 (DEFAULT_TABLE table=default.Table_2c99afea2b6e4963985dab32b545779e [id=e6cd13b44efa4cbbbdd2c14d8ea22267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.158881 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 025365d2fa0a43dd983d395a05cc2af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.161201 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdc9d47f2232403fa71e14a6fe6f33c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.162832 32355 tablet_service.cc:1468] Processing CreateTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 (DEFAULT_TABLE table=default.Table_4550e74cc6e845719c1466d3acd1b723 [id=587d6896c7dd40838ce8ad01026379a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.163018 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a0b9402a264800a1ea39707c10da0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.163144 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c8d480b8ba49a1be140b0e0a79f4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.164620 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ed4cb252b1452087f8ee53ec78d985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.166985 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f62ec83c88e43438b963e740d2c7004. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.168737 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 04eda92e14144733ab71a577dd3bd2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.169243 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 228375592aee48a1aa21e1d16962b663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.170994 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 8dc696236a15420e9666cca708068fb2 (DEFAULT_TABLE table=default.Table_f81a1165815543b781251c1ecfd06779 [id=491b95490f8c45eda181ea0cafd42368]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.173625 32493 tablet_service.cc:1468] Processing CreateTablet for tablet f279b63b55c64e548ded27b02025c9f5 (DEFAULT_TABLE table=default.Table_6e40e423312e4f9e95174f277b8897ce [id=7038d3807ece414f8a8e76bfe107c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.173980 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f279b63b55c64e548ded27b02025c9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.174136 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dc696236a15420e9666cca708068fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.181928 339 ts_tablet_manager.cc:1428] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.090s user 0.015s sys 0.007s
I20250623 08:04:27.182257 339 tablet_bootstrap.cc:492] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:27.183121 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_6bb7361aabf7440589a3958da502f36c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:27.183624 32130 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_9f3d2a1f5b3c41fc888725e2b7d45e0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.183751 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bb7361aabf7440589a3958da502f36c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.184230 339 tablet_bootstrap.cc:654] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.184155 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_d3f72251faee462c9c301b8aff464bb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.184623 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3f72251faee462c9c301b8aff464bb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:27.185469 32130 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f3d2a1f5b3c41fc888725e2b7d45e0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.185672 32126 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_7f123c9cf6f54d22ae18a9d6ce7d9d9c" } new_table_name: "default.Table_71fd2906f6b0404580099cb500e1f7fb" modify_external_catalogs: true
I20250623 08:04:27.186805 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 162308da898d4dde8ab5b1ab01d0f393 (DEFAULT_TABLE table=default.Table_c30b48097bd643b18b2a744839780b9e [id=7e59ad2235c1418d97f2b73ff2a48300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.187208 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162308da898d4dde8ab5b1ab01d0f393. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.192162 32629 tablet_service.cc:1468] Processing CreateTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 (DEFAULT_TABLE table=default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.192907 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ed4cb252b1452087f8ee53ec78d985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.196785 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 136e482995a24c5daec55a701f3091fa (DEFAULT_TABLE table=default.Table_fae6ac5a256346149e8cf56ae9dbdf3b [id=3c89160b4fb54f649fd3d222f6a6699d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.197280 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 136e482995a24c5daec55a701f3091fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.197741 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 136e482995a24c5daec55a701f3091fa (DEFAULT_TABLE table=default.Table_fae6ac5a256346149e8cf56ae9dbdf3b [id=3c89160b4fb54f649fd3d222f6a6699d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.199257 32632 tablet_service.cc:1468] Processing CreateTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 (DEFAULT_TABLE table=default.Table_4550e74cc6e845719c1466d3acd1b723 [id=587d6896c7dd40838ce8ad01026379a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.201189 32488 tablet_service.cc:1468] Processing CreateTablet for tablet 136e482995a24c5daec55a701f3091fa (DEFAULT_TABLE table=default.Table_fae6ac5a256346149e8cf56ae9dbdf3b [id=3c89160b4fb54f649fd3d222f6a6699d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.201591 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 136e482995a24c5daec55a701f3091fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.207850 342 tablet_bootstrap.cc:492] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.208016 342 ts_tablet_manager.cc:1397] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.010s
I20250623 08:04:27.208829 342 raft_consensus.cc:357] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.209071 342 raft_consensus.cc:383] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.209154 342 raft_consensus.cc:738] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.209421 342 consensus_queue.cc:260] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.209897 342 ts_tablet_manager.cc:1428] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:27.210202 342 tablet_bootstrap.cc:492] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.210923 32129 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_84c686e72347490ebc29749f8c9352f9" } modify_external_catalogs: true
I20250623 08:04:27.211133 32129 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_84c686e72347490ebc29749f8c9352f9" } modify_external_catalogs: true
I20250623 08:04:27.214134 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 136e482995a24c5daec55a701f3091fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.217861 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c8d480b8ba49a1be140b0e0a79f4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.191912 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 205da880685b49b3b8669f73dc2c563b (DEFAULT_TABLE table=default.Table_4d4535cab44545839eb3a559385e395a [id=bb849e86d7084959830c68207fb99061]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.219882 342 tablet_bootstrap.cc:654] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.220408 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 205da880685b49b3b8669f73dc2c563b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.221611 587 raft_consensus.cc:491] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.221992 587 raft_consensus.cc:513] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.222661 587 leader_election.cc:290] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:27.223872 32589 leader_election.cc:343] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 04a3ae58d26142d58c4df4f52d7e3e2c
I20250623 08:04:27.225668 32125 master_service.cc:948] ReplaceTablet: received request to replace tablet ae7c2ac546114a819a543f1667fb7844 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:27.226145 32125 master_service.cc:948] ReplaceTablet: received request to replace tablet 1e47b10f46b8402e94354ebca573c194 from {username='slave'} at 127.0.0.1:50426
W20250623 08:04:27.226539 32589 leader_election.cc:343] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 04a3ae58d26142d58c4df4f52d7e3e2c
I20250623 08:04:27.226903 32589 leader_election.cc:304] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.227700 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.227733 587 raft_consensus.cc:2747] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:27.233929 32124 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cde7df7d39a54114a4dcb28cfd1eb429 consensus state has no leader
I20250623 08:04:27.235955 32124 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f81a1165815543b781251c1ecfd06779" } modify_external_catalogs: true
I20250623 08:04:27.236202 32124 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_f81a1165815543b781251c1ecfd06779" } modify_external_catalogs: true
I20250623 08:04:27.240116 587 raft_consensus.cc:491] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.240291 587 raft_consensus.cc:513] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.240928 587 leader_election.cc:290] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:27.243106 597 raft_consensus.cc:491] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.243269 597 raft_consensus.cc:513] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.243837 597 leader_election.cc:290] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:27.245218 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "228375592aee48a1aa21e1d16962b663" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:27.245402 32589 leader_election.cc:343] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: c7385d86316f4a3ab68a77d18080ec20
W20250623 08:04:27.245620 32589 leader_election.cc:343] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: c7385d86316f4a3ab68a77d18080ec20
I20250623 08:04:27.245658 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "228375592aee48a1aa21e1d16962b663" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:27.245723 32589 leader_election.cc:304] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:27.246479 32453 leader_election.cc:343] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:27.247021 32453 leader_election.cc:343] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.247150 32453 leader_election.cc:304] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.247617 597 raft_consensus.cc:2747] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.250057 340 ts_tablet_manager.cc:1918] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.252302 32121 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 175ee90133994154b7e4b58b0770db3b
I20250623 08:04:27.252843 32121 catalog_manager.cc:6652] ReplaceTablet: tablet 175ee90133994154b7e4b58b0770db3b deleted and replaced by tablet e602d391526d4b0eb0dbf5189afbe075
I20250623 08:04:27.253886 32121 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_7e39ecbac4a8433e8ae98b0db22c30ea [id=8b20894a91c54a3eb08d5e1b4a42d269]). Expected version 2 got 0
I20250623 08:04:27.254869 587 raft_consensus.cc:2747] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.254977 32122 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 4cb66c7a273a412db3ae77af9e30d036
I20250623 08:04:27.255700 597 raft_consensus.cc:491] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.255677 32122 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_ec93cb7b48b247d08d3832d2589c93b7" } new_table_name: "default.Table_0ac52a7e208745e980f3a49c46b3cdb8" modify_external_catalogs: true
I20250623 08:04:27.255842 597 raft_consensus.cc:513] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.256449 597 leader_election.cc:290] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:27.257300 32127 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_3878732ceac64ef6bc3f0d170ffbe12a" } new_table_name: "default.Table_c390976ce10f40d193a81eae44cd862f" modify_external_catalogs: true
I20250623 08:04:27.257725 597 raft_consensus.cc:491] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.257862 597 raft_consensus.cc:513] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:27.258422 597 leader_election.cc:290] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:27.258621 32126 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31ecc74317d449e5b557b07a4fd14895 consensus state has no leader
I20250623 08:04:27.258948 32129 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 966a159e212040109e2842c15b69fb71
I20250623 08:04:27.259109 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fea0b5aa0c1403fb3ef3ab9ab040035" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.259654 32129 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_f1d8f7ef7cf144eaa1fd741b10eac3d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.259838 32453 leader_election.cc:343] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:27.260169 32129 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1d8f7ef7cf144eaa1fd741b10eac3d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:27.261330 32453 leader_election.cc:343] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 7fea0b5aa0c1403fb3ef3ab9ab040035
I20250623 08:04:27.261402 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c01121f2e644d6aa6e8ec228bf75be9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.261466 32453 leader_election.cc:304] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.261587 340 log.cc:1199] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/696f5d2797ef463bb72a2119501c6ef3
I20250623 08:04:27.261817 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c01121f2e644d6aa6e8ec228bf75be9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:27.261977 597 raft_consensus.cc:2747] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.262022 340 ts_tablet_manager.cc:1939] T 696f5d2797ef463bb72a2119501c6ef3 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:27.262145 342 tablet_bootstrap.cc:492] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.262318 342 ts_tablet_manager.cc:1397] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.007s
W20250623 08:04:27.262387 32453 leader_election.cc:343] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:27.262655 32453 leader_election.cc:343] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.262688 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet e22f9bae5f7248af99cec06ce6dbbd41
I20250623 08:04:27.262881 32453 leader_election.cc:304] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.263180 342 raft_consensus.cc:357] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.263280 597 raft_consensus.cc:2747] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.263309 32128 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_4d4535cab44545839eb3a559385e395a" } modify_external_catalogs: true
I20250623 08:04:27.263394 342 raft_consensus.cc:383] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.263492 342 raft_consensus.cc:738] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.263482 32128 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_4d4535cab44545839eb3a559385e395a" } modify_external_catalogs: true
I20250623 08:04:27.263695 339 tablet_bootstrap.cc:492] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:27.263728 342 consensus_queue.cc:260] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.263824 339 ts_tablet_manager.cc:1397] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.000s
I20250623 08:04:27.264223 342 ts_tablet_manager.cc:1428] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:27.264520 342 tablet_bootstrap.cc:492] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.265412 32126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_de3536331f614b4a8de796d3328c9fa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:27.265483 339 raft_consensus.cc:357] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.265647 339 raft_consensus.cc:383] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.265739 339 raft_consensus.cc:738] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:27.265915 32126 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de3536331f614b4a8de796d3328c9fa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.265966 339 consensus_queue.cc:260] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.266644 342 tablet_bootstrap.cc:654] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.267364 339 ts_tablet_manager.cc:1428] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:27.267640 339 tablet_bootstrap.cc:492] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:27.254683 32121 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_7c8c0d6c2a854f93845730b012a19628"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:27.270471 339 tablet_bootstrap.cc:654] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:27.270537 32121 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c8c0d6c2a854f93845730b012a19628 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.271739 340 tablet_replica.cc:331] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:27.271977 340 raft_consensus.cc:2241] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.272192 340 raft_consensus.cc:2270] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.277108 340 ts_tablet_manager.cc:1905] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.277771 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 696f5d2797ef463bb72a2119501c6ef3 (table default.Table_8a1e4cf5c8a54882b63f1dfd10533986 [id=f9f9b316ab75478ca20caa217c01f3b4]) successfully deleted
W20250623 08:04:27.278017 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 2060a643d0da47efa868f52bdb331b1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2060a643d0da47efa868f52bdb331b1f already in progress: creating tablet
I20250623 08:04:27.278584 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet 175ee90133994154b7e4b58b0770db3b with delete_type TABLET_DATA_DELETED (replaced by tablet e602d391526d4b0eb0dbf5189afbe075) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.279175 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet 966a159e212040109e2842c15b69fb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.279934 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 966a159e212040109e2842c15b69fb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.280045 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.280159 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.280305 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.280555 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 175ee90133994154b7e4b58b0770db3b with delete_type TABLET_DATA_DELETED (replaced by tablet e602d391526d4b0eb0dbf5189afbe075) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.280766 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.280963 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 966a159e212040109e2842c15b69fb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.281088 342 tablet_bootstrap.cc:492] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.281150 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.280820 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 175ee90133994154b7e4b58b0770db3b with delete_type TABLET_DATA_DELETED (replaced by tablet e602d391526d4b0eb0dbf5189afbe075) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.281231 342 ts_tablet_manager.cc:1397] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20250623 08:04:27.281628 32630 tablet_service.cc:1468] Processing CreateTablet for tablet f279b63b55c64e548ded27b02025c9f5 (DEFAULT_TABLE table=default.Table_6e40e423312e4f9e95174f277b8897ce [id=7038d3807ece414f8a8e76bfe107c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.282044 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f279b63b55c64e548ded27b02025c9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.281966 342 raft_consensus.cc:357] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.282229 342 raft_consensus.cc:383] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.282346 342 raft_consensus.cc:738] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.282593 342 consensus_queue.cc:260] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.283087 342 ts_tablet_manager.cc:1428] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:27.283371 342 tablet_bootstrap.cc:492] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.283717 587 raft_consensus.cc:491] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.283885 587 raft_consensus.cc:513] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.284539 587 leader_election.cc:290] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:27.284909 342 tablet_bootstrap.cc:654] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.285023 597 raft_consensus.cc:491] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.285197 597 raft_consensus.cc:513] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.285774 597 leader_election.cc:290] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:27.286199 32589 leader_election.cc:343] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 2eb99713f9214e6488d5c6e140ac263c
I20250623 08:04:27.286450 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "04eda92e14144733ab71a577dd3bd2b0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:27.287072 32453 leader_election.cc:343] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:27.287298 32589 leader_election.cc:343] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 2eb99713f9214e6488d5c6e140ac263c
I20250623 08:04:27.287433 32589 leader_election.cc:304] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.287956 587 raft_consensus.cc:2747] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:27.288218 32453 leader_election.cc:343] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 04eda92e14144733ab71a577dd3bd2b0
I20250623 08:04:27.288342 32453 leader_election.cc:304] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.288720 597 raft_consensus.cc:2747] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.290202 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.291000 340 ts_tablet_manager.cc:1918] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.291133 340 log.cc:1199] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/16d6b1e81e4d438f85d8fee4dd7d8b7d
I20250623 08:04:27.292974 344 ts_tablet_manager.cc:1918] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.293493 344 log.cc:1199] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/d67a8f1f570741ebb886c4d136ca40d2
I20250623 08:04:27.294024 344 ts_tablet_manager.cc:1939] T d67a8f1f570741ebb886c4d136ca40d2 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:27.295711 342 tablet_bootstrap.cc:492] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.295866 342 ts_tablet_manager.cc:1397] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250623 08:04:27.296069 32130 master_service.cc:948] ReplaceTablet: received request to replace tablet 2060a643d0da47efa868f52bdb331b1f from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:27.296643 342 raft_consensus.cc:357] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.296815 342 raft_consensus.cc:383] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.296893 342 raft_consensus.cc:738] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.297093 342 consensus_queue.cc:260] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.297580 342 ts_tablet_manager.cc:1428] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:27.297870 342 tablet_bootstrap.cc:492] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.299458 342 tablet_bootstrap.cc:654] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.299885 32130 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_d000f5be42824b9fad2bdeb33231b5f8" } new_table_name: "default.Table_91b12dbe72c64d16a3334faf46445848" modify_external_catalogs: true
I20250623 08:04:27.301368 32123 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_79be3d7247284828a82f02d96faa60cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.301916 32123 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79be3d7247284828a82f02d96faa60cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.302476 343 tablet_bootstrap.cc:654] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.303443 341 raft_consensus.cc:2270] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.304000 341 ts_tablet_manager.cc:1905] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.304080 342 tablet_bootstrap.cc:492] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.304207 342 ts_tablet_manager.cc:1397] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250623 08:04:27.304857 342 raft_consensus.cc:357] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.305042 342 raft_consensus.cc:383] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.305109 342 raft_consensus.cc:738] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.305316 342 consensus_queue.cc:260] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.305821 342 ts_tablet_manager.cc:1428] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:27.300614 339 tablet_bootstrap.cc:492] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:27.306113 339 ts_tablet_manager.cc:1397] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250623 08:04:27.306901 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet d67a8f1f570741ebb886c4d136ca40d2 (table default.Table_be622804eb114901b245383926c3afec [id=b020e49a601748e9bcbf397e235dc596]) successfully deleted
I20250623 08:04:27.306877 339 raft_consensus.cc:357] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.307080 339 raft_consensus.cc:383] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.307164 339 raft_consensus.cc:738] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.307437 339 consensus_queue.cc:260] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:27.308120 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 4ea81642050141a9be84f151a5785abc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea81642050141a9be84f151a5785abc already in progress: creating tablet
I20250623 08:04:27.309178 339 ts_tablet_manager.cc:1428] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:27.309504 339 tablet_bootstrap.cc:492] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:27.311038 342 tablet_bootstrap.cc:492] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.312609 342 tablet_bootstrap.cc:654] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.313195 32124 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 8dc696236a15420e9666cca708068fb2
I20250623 08:04:27.313956 32124 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_1671bbefe7bd4d4b86a917373288a1a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.314495 32124 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1671bbefe7bd4d4b86a917373288a1a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:27.315681 32122 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ac9f38877b24d619c280a1aaa0b7b08 consensus state has no leader
I20250623 08:04:27.316016 32122 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_c7119eddeaf44e1b89d2dbd1e38151f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.316493 32122 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.317598 339 tablet_bootstrap.cc:654] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:27.317589 32127 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 283bebc7063d437ba3e541d7dee3c0b5 consensus state has no leader
I20250623 08:04:27.317915 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_4a19c65595654fa98efc42626b6a9d85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.318327 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a19c65595654fa98efc42626b6a9d85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.318804 342 tablet_bootstrap.cc:492] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.318933 342 ts_tablet_manager.cc:1397] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20250623 08:04:27.318995 340 ts_tablet_manager.cc:1939] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:27.319231 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 205da880685b49b3b8669f73dc2c563b
I20250623 08:04:27.319676 342 raft_consensus.cc:357] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.319869 342 raft_consensus.cc:383] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.319954 342 raft_consensus.cc:738] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.320178 342 consensus_queue.cc:260] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.320609 342 ts_tablet_manager.cc:1428] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:27.320902 342 tablet_bootstrap.cc:492] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.322829 342 tablet_bootstrap.cc:654] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.331463 344 tablet_replica.cc:331] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:27.331722 344 raft_consensus.cc:2241] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.331957 344 raft_consensus.cc:2270] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.332521 344 ts_tablet_manager.cc:1905] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.335011 32126 master_service.cc:948] ReplaceTablet: received request to replace tablet d0ee5a4fe03848229524cc0105226d8c from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:27.337872 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dc696236a15420e9666cca708068fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.338693 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dc696236a15420e9666cca708068fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.340188 343 tablet_bootstrap.cc:492] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.340175 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dc696236a15420e9666cca708068fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.340452 343 ts_tablet_manager.cc:1397] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.255s user 0.004s sys 0.000s
I20250623 08:04:27.340507 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 205da880685b49b3b8669f73dc2c563b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.341348 343 raft_consensus.cc:357] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.341624 343 raft_consensus.cc:383] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.341770 343 raft_consensus.cc:738] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.342080 343 consensus_queue.cc:260] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.342646 339 tablet_bootstrap.cc:492] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:27.342819 339 ts_tablet_manager.cc:1397] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20250623 08:04:27.343636 339 raft_consensus.cc:357] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.343803 339 raft_consensus.cc:383] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.343883 339 raft_consensus.cc:738] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.344136 339 consensus_queue.cc:260] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.345245 343 ts_tablet_manager.cc:1428] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:27.345587 343 tablet_bootstrap.cc:492] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.349031 339 ts_tablet_manager.cc:1428] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:27.351859 343 tablet_bootstrap.cc:654] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.353451 339 tablet_bootstrap.cc:492] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:27.354409 341 ts_tablet_manager.cc:1918] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.355211 339 tablet_bootstrap.cc:654] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.356827 341 log.cc:1199] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b6ad6cd629c4423ca997efb517cc975b
I20250623 08:04:27.357831 341 ts_tablet_manager.cc:1939] T b6ad6cd629c4423ca997efb517cc975b P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.360867 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 169) took 1071 ms. Trace:
I20250623 08:04:27.357434 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 205da880685b49b3b8669f73dc2c563b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.361183 341 rpcz_store.cc:276] 0623 08:04:26.289854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.290509 (+ 655us) service_pool.cc:225] Handling call
0623 08:04:27.118404 (+827895us) ts_tablet_manager.cc:1145] Deleting tablet b6ad6cd629c4423ca997efb517cc975b
0623 08:04:27.118414 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:27.336690 (+218276us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:27.354275 (+ 17585us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:27.360814 (+ 6539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":827528}
I20250623 08:04:27.361783 341 tablet_replica.cc:331] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.362032 341 raft_consensus.cc:2241] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:27.362473 341 pending_rounds.cc:70] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:27.364878 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d (table default.Table_c4287575a32b4847ba2379ef60074ba9 [id=abda89a30971466ea9a1f410d424ef58]) successfully deleted
I20250623 08:04:27.364868 341 pending_rounds.cc:77] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727389181190144 op_type: NO_OP noop_request { }
I20250623 08:04:27.365144 341 raft_consensus.cc:2887] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20250623 08:04:27.365149 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet ae7c2ac546114a819a543f1667fb7844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae7c2ac546114a819a543f1667fb7844 already in progress: creating tablet
W20250623 08:04:27.365345 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 279624b8bb834b95a4f08045968ebfb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 279624b8bb834b95a4f08045968ebfb4 already in progress: creating tablet
I20250623 08:04:27.365339 341 raft_consensus.cc:2270] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:27.365556 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 417a050eada4403ebbddff255fb6fb5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417a050eada4403ebbddff255fb6fb5a already in progress: creating tablet
W20250623 08:04:27.365784 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet b980c746a3c042bfae90aa076acbb1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b980c746a3c042bfae90aa076acbb1df already in progress: creating tablet
W20250623 08:04:27.366008 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 79731106915d4cebb9dcc3c10386ffa9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79731106915d4cebb9dcc3c10386ffa9 already in progress: creating tablet
W20250623 08:04:27.366236 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 175ee90133994154b7e4b58b0770db3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 175ee90133994154b7e4b58b0770db3b already in progress: creating tablet
W20250623 08:04:27.366447 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 4cb66c7a273a412db3ae77af9e30d036 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cb66c7a273a412db3ae77af9e30d036 already in progress: creating tablet
I20250623 08:04:27.366698 339 tablet_bootstrap.cc:492] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
W20250623 08:04:27.366688 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 966a159e212040109e2842c15b69fb71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 966a159e212040109e2842c15b69fb71 already in progress: creating tablet
I20250623 08:04:27.366904 339 ts_tablet_manager.cc:1397] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
W20250623 08:04:27.366926 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet e22f9bae5f7248af99cec06ce6dbbd41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e22f9bae5f7248af99cec06ce6dbbd41 already in progress: creating tablet
I20250623 08:04:27.366907 341 ts_tablet_manager.cc:1905] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:27.367110 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 8dc696236a15420e9666cca708068fb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dc696236a15420e9666cca708068fb2 already in progress: creating tablet
W20250623 08:04:27.367319 32114 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 205da880685b49b3b8669f73dc2c563b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 205da880685b49b3b8669f73dc2c563b already in progress: creating tablet
I20250623 08:04:27.367653 339 raft_consensus.cc:357] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.367854 339 raft_consensus.cc:383] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.367934 339 raft_consensus.cc:738] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.368201 597 raft_consensus.cc:491] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.368498 597 raft_consensus.cc:513] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.369179 597 leader_election.cc:290] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:27.369367 597 raft_consensus.cc:491] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.369330 339 consensus_queue.cc:260] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.369513 597 raft_consensus.cc:513] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.370059 597 leader_election.cc:290] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:27.370822 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 205da880685b49b3b8669f73dc2c563b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.371349 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet ce1f3ef3690c4837b95b105733884f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.371887 342 tablet_bootstrap.cc:492] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.371997 343 tablet_bootstrap.cc:492] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.372051 342 ts_tablet_manager.cc:1397] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.009s
I20250623 08:04:27.372282 343 ts_tablet_manager.cc:1397] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20250623 08:04:27.372542 339 ts_tablet_manager.cc:1428] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:27.372829 342 raft_consensus.cc:357] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.373013 342 raft_consensus.cc:383] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.373103 342 raft_consensus.cc:738] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.373274 343 raft_consensus.cc:357] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:27.373332 342 consensus_queue.cc:260] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.373637 343 raft_consensus.cc:383] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.373762 343 raft_consensus.cc:738] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.373831 342 ts_tablet_manager.cc:1428] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:27.374373 342 tablet_bootstrap.cc:492] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.374586 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef3690c4837b95b105733884f1f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.374644 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2414d24ab6c54f36bb1e554f7231a5d7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.374892 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2414d24ab6c54f36bb1e554f7231a5d7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:27.375025 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef3690c4837b95b105733884f1f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:27.375079 32379 raft_consensus.cc:2466] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:27.375237 32381 raft_consensus.cc:2466] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:27.375316 32453 leader_election.cc:343] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.375589 339 tablet_bootstrap.cc:492] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:27.375550 32453 leader_election.cc:304] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:27.375984 32453 leader_election.cc:304] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:27.376781 597 raft_consensus.cc:2802] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:27.376842 342 tablet_bootstrap.cc:654] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.376916 597 raft_consensus.cc:491] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:27.377023 597 raft_consensus.cc:3058] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.377909 657 raft_consensus.cc:2802] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250623 08:04:27.377998 32453 leader_election.cc:343] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.378072 657 raft_consensus.cc:491] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:27.378216 657 raft_consensus.cc:3058] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.374120 343 consensus_queue.cc:260] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:27.380820 343 ts_tablet_manager.cc:1428] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250623 08:04:27.381174 343 tablet_bootstrap.cc:492] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.381899 657 raft_consensus.cc:513] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.383601 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef3690c4837b95b105733884f1f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:27.384132 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef3690c4837b95b105733884f1f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:27.384320 32381 raft_consensus.cc:3058] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.385409 343 tablet_bootstrap.cc:654] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.389123 32381 raft_consensus.cc:2466] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:27.390157 339 tablet_bootstrap.cc:654] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.399036 341 ts_tablet_manager.cc:1918] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:27.399410 341 log.cc:1199] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/0c77fd6e54904722acb2d1437c627ba8
I20250623 08:04:27.400068 341 ts_tablet_manager.cc:1939] T 0c77fd6e54904722acb2d1437c627ba8 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.401386 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 167) took 1111 ms. Trace:
I20250623 08:04:27.405081 339 tablet_bootstrap.cc:492] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:27.405234 339 ts_tablet_manager.cc:1397] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20250623 08:04:27.402949 343 tablet_bootstrap.cc:492] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.405516 343 ts_tablet_manager.cc:1397] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20250623 08:04:27.406250 339 raft_consensus.cc:357] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.406443 339 raft_consensus.cc:383] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.406316 343 raft_consensus.cc:357] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:27.406551 339 raft_consensus.cc:738] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.406683 343 raft_consensus.cc:383] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.406852 343 raft_consensus.cc:738] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.407146 343 consensus_queue.cc:260] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:27.407456 339 consensus_queue.cc:260] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.409978 343 ts_tablet_manager.cc:1428] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250623 08:04:27.410293 343 tablet_bootstrap.cc:492] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.412256 343 tablet_bootstrap.cc:654] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.405227 341 rpcz_store.cc:276] 0623 08:04:26.289590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.290737 (+ 1147us) service_pool.cc:225] Handling call
0623 08:04:27.361706 (+1070969us) ts_tablet_manager.cc:1145] Deleting tablet 0c77fd6e54904722acb2d1437c627ba8
0623 08:04:27.361715 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:27.392442 (+ 30727us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:27.395455 (+ 3013us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:27.401317 (+ 5862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1070464}
I20250623 08:04:27.413156 341 tablet_replica.cc:331] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.413389 341 raft_consensus.cc:2241] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.413609 341 raft_consensus.cc:2270] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:27.413998 32453 leader_election.cc:343] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.414340 339 ts_tablet_manager.cc:1428] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250623 08:04:27.414633 32453 leader_election.cc:304] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:27.415001 657 leader_election.cc:290] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:27.415158 657 raft_consensus.cc:491] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.415452 342 tablet_bootstrap.cc:492] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.415346 657 raft_consensus.cc:513] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.415724 341 ts_tablet_manager.cc:1905] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.415987 657 leader_election.cc:290] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:27.416167 657 raft_consensus.cc:2802] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:27.416293 657 raft_consensus.cc:695] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:27.416606 657 consensus_queue.cc:237] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.417694 597 raft_consensus.cc:513] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.418244 597 leader_election.cc:290] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:27.418257 339 tablet_bootstrap.cc:492] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:27.419201 587 raft_consensus.cc:491] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.419394 587 raft_consensus.cc:513] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.420051 339 tablet_bootstrap.cc:654] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.420066 587 leader_election.cc:290] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:27.421072 32589 leader_election.cc:343] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 69e772df590747ff841be5f218e55acf
I20250623 08:04:27.422434 342 ts_tablet_manager.cc:1397] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.008s
I20250623 08:04:27.423782 339 tablet_bootstrap.cc:492] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:27.423978 339 ts_tablet_manager.cc:1397] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250623 08:04:27.424743 339 raft_consensus.cc:357] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.424933 339 raft_consensus.cc:383] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.425036 339 raft_consensus.cc:738] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.424970 342 raft_consensus.cc:357] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.425125 342 raft_consensus.cc:383] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.425221 342 raft_consensus.cc:738] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.425298 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "696f5d2797ef463bb72a2119501c6ef3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.425349 339 consensus_queue.cc:260] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.425422 342 consensus_queue.cc:260] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.425940 342 ts_tablet_manager.cc:1428] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:27.426231 342 tablet_bootstrap.cc:492] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.426510 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2414d24ab6c54f36bb1e554f7231a5d7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:27.426717 339 ts_tablet_manager.cc:1428] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250623 08:04:27.427497 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2414d24ab6c54f36bb1e554f7231a5d7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:27.427675 32379 raft_consensus.cc:3058] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:27.428298 32453 leader_election.cc:343] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 696f5d2797ef463bb72a2119501c6ef3
W20250623 08:04:27.428563 32453 leader_election.cc:343] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.428664 32453 leader_election.cc:304] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
W20250623 08:04:27.429021 32453 leader_election.cc:343] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:27.429425 32589 leader_election.cc:343] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 69e772df590747ff841be5f218e55acf
I20250623 08:04:27.429533 32589 leader_election.cc:304] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.430275 32379 raft_consensus.cc:2466] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:27.430667 662 raft_consensus.cc:2747] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.431908 32453 leader_election.cc:304] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:27.432247 662 raft_consensus.cc:2802] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:27.432425 339 tablet_bootstrap.cc:492] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:27.432461 662 raft_consensus.cc:695] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:27.432854 662 consensus_queue.cc:237] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.434131 339 tablet_bootstrap.cc:654] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.434353 587 raft_consensus.cc:2747] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.435703 32126 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_b7f7dd030adc46329a94b5de210eb2f9" } new_table_name: "default.Table_cc3ded2eef374e8d9f476cc87bfc2470" modify_external_catalogs: true
I20250623 08:04:27.437579 342 tablet_bootstrap.cc:654] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.438669 343 tablet_bootstrap.cc:492] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.438953 343 ts_tablet_manager.cc:1397] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.001s
I20250623 08:04:27.440002 343 raft_consensus.cc:357] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.441283 343 raft_consensus.cc:383] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.441422 343 raft_consensus.cc:738] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.441730 343 consensus_queue.cc:260] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.443298 343 ts_tablet_manager.cc:1428] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:27.443652 343 tablet_bootstrap.cc:492] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.447883 342 tablet_bootstrap.cc:492] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.448067 342 ts_tablet_manager.cc:1397] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250623 08:04:27.448874 342 raft_consensus.cc:357] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.449082 342 raft_consensus.cc:383] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.449172 342 raft_consensus.cc:738] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.449406 342 consensus_queue.cc:260] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.449882 342 ts_tablet_manager.cc:1428] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:27.450054 343 tablet_bootstrap.cc:654] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.450176 342 tablet_bootstrap.cc:492] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.452071 342 tablet_bootstrap.cc:654] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.457842 32128 master_service.cc:948] ReplaceTablet: received request to replace tablet 7300f526d91f441bb164fe47f3a4f0ce from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:27.462405 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b6ad6cd629c4423ca997efb517cc975b (table default.Table_47264042125847ac997923572c293a3e [id=4214b1f0ca1744dc9b46d3f4d7618ba3]) successfully deleted
I20250623 08:04:27.462664 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 0c77fd6e54904722acb2d1437c627ba8 (table default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]) successfully deleted
W20250623 08:04:27.464226 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet ea702abb31fb41498ab61bbb05b1f7e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea702abb31fb41498ab61bbb05b1f7e2 already in progress: creating tablet
I20250623 08:04:27.465288 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 69e772df590747ff841be5f218e55acf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:23 UTC) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.465546 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 with delete_type TABLET_DATA_DELETED (replaced by tablet d02f74c24a174b18b08a870a4e47fefb) from {username='slave'} at 127.0.0.1:36600
W20250623 08:04:27.473057 32130 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f has no consensus state
I20250623 08:04:27.473455 32130 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c30b48097bd643b18b2a744839780b9e" } modify_external_catalogs: true
I20250623 08:04:27.473650 32130 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_c30b48097bd643b18b2a744839780b9e" } modify_external_catalogs: true
I20250623 08:04:27.474522 32123 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a4ca66fa08cc441ab5571f9968f30914" } modify_external_catalogs: true
I20250623 08:04:27.474690 32123 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a4ca66fa08cc441ab5571f9968f30914" } modify_external_catalogs: true
I20250623 08:04:27.475508 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet 205da880685b49b3b8669f73dc2c563b from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:27.475993 32122 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_e7f8192c00524b5899152fa3f717b898" } modify_external_catalogs: true
I20250623 08:04:27.476136 32122 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_e7f8192c00524b5899152fa3f717b898" } modify_external_catalogs: true
I20250623 08:04:27.476830 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_9fc75fd22c6a467782fe3b4c3b123602"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.477357 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fc75fd22c6a467782fe3b4c3b123602 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.478855 341 ts_tablet_manager.cc:1918] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.481760 341 log.cc:1199] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b9d871507dcc46fc80f6e0ac739c6476
I20250623 08:04:27.482362 341 ts_tablet_manager.cc:1939] T b9d871507dcc46fc80f6e0ac739c6476 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.483278 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 168) took 1193 ms. Trace:
I20250623 08:04:27.483481 341 rpcz_store.cc:276] 0623 08:04:26.289749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.291072 (+ 1323us) service_pool.cc:225] Handling call
0623 08:04:27.413080 (+1122008us) ts_tablet_manager.cc:1145] Deleting tablet b9d871507dcc46fc80f6e0ac739c6476
0623 08:04:27.413089 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:27.427034 (+ 13945us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:27.478688 (+ 51654us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:27.483241 (+ 4553us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1121736}
I20250623 08:04:27.484666 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 166) took 1194 ms. Trace:
I20250623 08:04:27.484813 341 rpcz_store.cc:276] 0623 08:04:26.289430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.291372 (+ 1942us) service_pool.cc:225] Handling call
0623 08:04:27.483988 (+1192616us) ts_tablet_manager.cc:1145] Deleting tablet d4221d7500d1458db75926d7b274a654
0623 08:04:27.483997 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:27.484112 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191895}
I20250623 08:04:27.485314 341 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36600 (request call id 171) took 1165 ms. Trace:
I20250623 08:04:27.486296 341 rpcz_store.cc:276] 0623 08:04:26.320289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.320340 (+ 51us) service_pool.cc:225] Handling call
0623 08:04:27.485184 (+1164844us) ts_tablet_manager.cc:1145] Deleting tablet 35ab9f77a81a4490abb0d2c0c69cd90f
0623 08:04:27.485192 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:27.485283 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164619}
I20250623 08:04:27.486207 343 tablet_bootstrap.cc:492] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.486868 343 ts_tablet_manager.cc:1397] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20250623 08:04:27.487646 343 raft_consensus.cc:357] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.488009 343 raft_consensus.cc:383] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.488126 343 raft_consensus.cc:738] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.488426 343 consensus_queue.cc:260] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.490139 343 ts_tablet_manager.cc:1428] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:27.490474 343 tablet_bootstrap.cc:492] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.494048 343 tablet_bootstrap.cc:654] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.494942 341 tablet_replica.cc:331] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.495987 341 raft_consensus.cc:2241] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.506914 341 raft_consensus.cc:2270] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.514245 610 raft_consensus.cc:491] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.514441 610 raft_consensus.cc:513] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:27.514020 32114 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 46130.2us
I20250623 08:04:27.515206 610 leader_election.cc:290] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:27.515694 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ac9f38877b24d619c280a1aaa0b7b08" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:27.516008 32520 raft_consensus.cc:2466] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:27.515137 341 ts_tablet_manager.cc:1905] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.517378 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ac9f38877b24d619c280a1aaa0b7b08" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:27.519910 32314 leader_election.cc:343] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.520426 32314 leader_election.cc:304] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:27.520709 610 raft_consensus.cc:2802] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:27.520813 610 raft_consensus.cc:491] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:27.520900 610 raft_consensus.cc:3058] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.522025 342 tablet_bootstrap.cc:492] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.522166 342 ts_tablet_manager.cc:1397] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.020s
I20250623 08:04:27.523005 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b9d871507dcc46fc80f6e0ac739c6476 (table default.Table_d27db7e0fea34b5182353dee8ae23cac [id=816747bd5e004ae5964b7b885aa249fb]) successfully deleted
I20250623 08:04:27.523087 342 raft_consensus.cc:357] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.523258 342 raft_consensus.cc:383] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 08:04:27.523308 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d4221d7500d1458db75926d7b274a654 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4221d7500d1458db75926d7b274a654 already in progress: creating tablet
I20250623 08:04:27.523350 342 raft_consensus.cc:738] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.523571 342 consensus_queue.cc:260] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.523792 610 raft_consensus.cc:513] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.524094 342 ts_tablet_manager.cc:1428] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:27.524802 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ac9f38877b24d619c280a1aaa0b7b08" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
W20250623 08:04:27.525904 32314 leader_election.cc:343] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.526788 610 leader_election.cc:290] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:27.528523 343 tablet_bootstrap.cc:492] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.528687 343 ts_tablet_manager.cc:1397] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
W20250623 08:04:27.528944 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 32c82141edea4068b54755a85c6d7fb8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32c82141edea4068b54755a85c6d7fb8 already in progress: creating tablet
W20250623 08:04:27.529196 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 19536c16ff32460e9db571e4ce411bab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19536c16ff32460e9db571e4ce411bab already in progress: creating tablet
W20250623 08:04:27.529407 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 7da84969a6464b5caeaf47af0cbaaa4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7da84969a6464b5caeaf47af0cbaaa4d already in progress: creating tablet
I20250623 08:04:27.529488 343 raft_consensus.cc:357] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:27.529639 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 4ea81642050141a9be84f151a5785abc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ea81642050141a9be84f151a5785abc already in progress: creating tablet
I20250623 08:04:27.529702 343 raft_consensus.cc:383] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.529781 343 raft_consensus.cc:738] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.530061 343 consensus_queue.cc:260] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.531724 343 ts_tablet_manager.cc:1428] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250623 08:04:27.532030 343 tablet_bootstrap.cc:492] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.521200 344 ts_tablet_manager.cc:1918] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.533445 344 log.cc:1199] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/7fea0b5aa0c1403fb3ef3ab9ab040035
I20250623 08:04:27.536401 344 ts_tablet_manager.cc:1939] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:27.538410 343 tablet_bootstrap.cc:654] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.538865 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ac9f38877b24d619c280a1aaa0b7b08" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:27.539048 342 tablet_bootstrap.cc:492] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.539098 32520 raft_consensus.cc:3058] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.542071 32520 raft_consensus.cc:2466] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:27.542824 32126 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50426:
table { table_name: "default.Table_a94b76793b7a49a687d0052141dd50c4" } new_table_name: "default.Table_a1f91329476d4e4f9be66462bda04dbf" modify_external_catalogs: true
I20250623 08:04:27.542929 32314 leader_election.cc:304] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:27.543416 610 raft_consensus.cc:2802] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:27.543560 610 raft_consensus.cc:695] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:27.543610 342 tablet_bootstrap.cc:654] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.543831 610 consensus_queue.cc:237] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.544319 32128 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 7300f526d91f441bb164fe47f3a4f0ce
I20250623 08:04:27.544780 32128 catalog_manager.cc:6652] ReplaceTablet: tablet 7300f526d91f441bb164fe47f3a4f0ce deleted and replaced by tablet 736eb56dae194df2af90bf33d3294a4e
I20250623 08:04:27.545135 32128 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_3115dd6f7dc948aa927f4ab714867992"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:27.545665 32128 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3115dd6f7dc948aa927f4ab714867992 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.546872 341 ts_tablet_manager.cc:1918] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.547021 341 log.cc:1199] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/ce1f3ef3690c4837b95b105733884f1f
I20250623 08:04:27.547128 32127 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50426:
name: "default.Table_7359680519754448a8464b0e8131bd55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:27.547377 341 ts_tablet_manager.cc:1939] T ce1f3ef3690c4837b95b105733884f1f P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:27.547629 32127 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7359680519754448a8464b0e8131bd55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:27.552415 341 tablet_replica.cc:331] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.552621 341 raft_consensus.cc:2241] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.552814 341 raft_consensus.cc:2270] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.553337 341 ts_tablet_manager.cc:1905] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.557667 32124 master_service.cc:948] ReplaceTablet: received request to replace tablet 2ecf4af9e476495683bd63cac20d4672 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:27.558800 32125 catalog_manager.cc:5582] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:27.559134 343 tablet_bootstrap.cc:492] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.559262 343 ts_tablet_manager.cc:1397] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20250623 08:04:27.560060 343 raft_consensus.cc:357] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.560241 343 raft_consensus.cc:383] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.560317 343 raft_consensus.cc:738] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.560508 343 consensus_queue.cc:260] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.563364 343 ts_tablet_manager.cc:1428] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:27.563761 343 tablet_bootstrap.cc:492] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.565869 343 tablet_bootstrap.cc:654] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.566836 32129 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 4ac9f38877b24d619c280a1aaa0b7b08 (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:27.567052 32129 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 0311ce67989f49e8866234a88e5b4d09 (table default.Table_e962b9945a5041039f7e33b98b653175 [id=9d9e232100264e378516c06c61310750]). Expected version 1 got 0
I20250623 08:04:27.568608 32129 master_service.cc:948] ReplaceTablet: received request to replace tablet cde7df7d39a54114a4dcb28cfd1eb429 from {username='slave'} at 127.0.0.1:50426
I20250623 08:04:27.569121 587 raft_consensus.cc:491] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.569312 587 raft_consensus.cc:513] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.569922 587 leader_election.cc:290] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:27.572518 32589 leader_election.cc:343] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: a46c0a89fc014054a75ed198e824b5b6
I20250623 08:04:27.572642 342 tablet_bootstrap.cc:492] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
W20250623 08:04:27.572722 32589 leader_election.cc:343] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: a46c0a89fc014054a75ed198e824b5b6
I20250623 08:04:27.572827 342 ts_tablet_manager.cc:1397] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20250623 08:04:27.572860 32589 leader_election.cc:304] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.573276 587 raft_consensus.cc:2747] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.573630 342 raft_consensus.cc:357] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.573848 342 raft_consensus.cc:383] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.573927 342 raft_consensus.cc:738] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.574167 342 consensus_queue.cc:260] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.575444 342 ts_tablet_manager.cc:1428] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:27.575858 342 tablet_bootstrap.cc:492] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.575963 341 ts_tablet_manager.cc:1918] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.576093 341 log.cc:1199] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/9c4dec1a48f84fb3bd7bbd757f7b0c1b
I20250623 08:04:27.576443 341 ts_tablet_manager.cc:1939] T 9c4dec1a48f84fb3bd7bbd757f7b0c1b P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.577754 342 tablet_bootstrap.cc:654] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.582814 662 raft_consensus.cc:491] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.582998 662 raft_consensus.cc:513] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.583871 341 tablet_replica.cc:331] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.583838 662 leader_election.cc:290] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:27.584098 341 raft_consensus.cc:2241] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.584281 341 raft_consensus.cc:2270] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.584746 343 tablet_bootstrap.cc:492] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.584868 341 ts_tablet_manager.cc:1905] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.584946 343 ts_tablet_manager.cc:1397] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20250623 08:04:27.584723 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ee5a4fe03848229524cc0105226d8c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.585423 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ee5a4fe03848229524cc0105226d8c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:27.585652 32379 raft_consensus.cc:2466] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:27.585958 343 raft_consensus.cc:357] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:27.586097 32453 leader_election.cc:343] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.586179 343 raft_consensus.cc:383] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.586287 343 raft_consensus.cc:738] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.586560 32453 leader_election.cc:304] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:27.586570 343 consensus_queue.cc:260] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.587205 662 raft_consensus.cc:2802] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:27.587320 662 raft_consensus.cc:491] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:27.587343 343 ts_tablet_manager.cc:1428] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250623 08:04:27.587401 662 raft_consensus.cc:3058] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.590119 662 raft_consensus.cc:513] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.591164 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ee5a4fe03848229524cc0105226d8c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:27.591388 32379 raft_consensus.cc:3058] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.591624 343 tablet_bootstrap.cc:492] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.593587 343 tablet_bootstrap.cc:654] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.594082 32379 raft_consensus.cc:2466] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:27.594607 32453 leader_election.cc:304] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:27.595104 657 raft_consensus.cc:2802] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:27.595286 657 raft_consensus.cc:695] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:27.595589 657 consensus_queue.cc:237] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.598055 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ee5a4fe03848229524cc0105226d8c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
W20250623 08:04:27.598623 32453 leader_election.cc:343] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.598861 662 leader_election.cc:290] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:27.601810 662 raft_consensus.cc:491] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.601979 662 raft_consensus.cc:513] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.603112 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ea81642050141a9be84f151a5785abc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:27.603339 32379 raft_consensus.cc:2466] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:27.603587 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ea81642050141a9be84f151a5785abc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.604295 32453 leader_election.cc:304] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:27.604624 659 raft_consensus.cc:2802] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250623 08:04:27.604652 32453 leader_election.cc:343] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.604729 659 raft_consensus.cc:491] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:27.604846 662 leader_election.cc:290] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:27.604912 659 raft_consensus.cc:3058] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.607195 341 ts_tablet_manager.cc:1918] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.607354 341 log.cc:1199] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/d67a8f1f570741ebb886c4d136ca40d2
I20250623 08:04:27.607723 341 ts_tablet_manager.cc:1939] T d67a8f1f570741ebb886c4d136ca40d2 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.608700 659 raft_consensus.cc:513] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.610378 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ea81642050141a9be84f151a5785abc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
W20250623 08:04:27.610954 32453 leader_election.cc:343] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.611310 659 leader_election.cc:290] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:27.611708 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ea81642050141a9be84f151a5785abc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:27.611907 32379 raft_consensus.cc:3058] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.614701 32379 raft_consensus.cc:2466] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:27.615397 32453 leader_election.cc:304] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:27.615938 659 raft_consensus.cc:2802] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:27.616642 659 raft_consensus.cc:695] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:27.617018 659 consensus_queue.cc:237] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.617401 341 tablet_replica.cc:331] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.617600 341 raft_consensus.cc:2241] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.617802 341 raft_consensus.cc:2270] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.618325 341 ts_tablet_manager.cc:1905] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:27.626314 32114 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 96318.8us
I20250623 08:04:27.629344 343 tablet_bootstrap.cc:492] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.629530 343 ts_tablet_manager.cc:1397] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.002s
I20250623 08:04:27.630515 343 raft_consensus.cc:357] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.630731 343 raft_consensus.cc:383] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.630851 343 raft_consensus.cc:738] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.635926 429 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:48186 (request call id 0) took 1171 ms. Trace:
I20250623 08:04:27.636049 343 consensus_queue.cc:260] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.636852 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 7300f526d91f441bb164fe47f3a4f0ce with delete_type TABLET_DATA_DELETED (replaced by tablet 736eb56dae194df2af90bf33d3294a4e) from {username='slave'} at 127.0.0.1:36600
I20250623 08:04:27.637708 343 ts_tablet_manager.cc:1428] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20250623 08:04:27.637995 343 tablet_bootstrap.cc:492] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.639001 560 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9752155c20ee42fbb7ff7b7d5b682444)
I20250623 08:04:27.639220 560 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:27.639285 618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9752155c20ee42fbb7ff7b7d5b682444. Latest consensus state: current_term: 2 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:27.639607 618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:27.642840 343 tablet_bootstrap.cc:654] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.642911 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 7300f526d91f441bb164fe47f3a4f0ce with delete_type TABLET_DATA_DELETED (replaced by tablet 736eb56dae194df2af90bf33d3294a4e) from {username='slave'} at 127.0.0.1:35526
I20250623 08:04:27.643226 340 tablet_replica.cc:331] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:27.643406 32069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 3 candidate_status { last_received { term: 2 index: 263 } } ignore_live_leader: false dest_uuid: "d31835ec4ad74332a00998d431492466" is_pre_election: true
I20250623 08:04:27.643463 340 raft_consensus.cc:2241] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.643713 340 raft_consensus.cc:2270] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.644261 340 ts_tablet_manager.cc:1905] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.644263 341 ts_tablet_manager.cc:1918] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.644394 341 log.cc:1199] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/35428260e05e447ebf88ea41e63cdd02
I20250623 08:04:27.644754 341 ts_tablet_manager.cc:1939] T 35428260e05e447ebf88ea41e63cdd02 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.645283 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 7300f526d91f441bb164fe47f3a4f0ce with delete_type TABLET_DATA_DELETED (replaced by tablet 736eb56dae194df2af90bf33d3294a4e) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.645716 341 tablet_replica.cc:331] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.645905 341 raft_consensus.cc:2241] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.646066 341 raft_consensus.cc:2270] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.646503 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d with delete_type TABLET_DATA_DELETED (replaced by tablet c7fcf583064f4538913d9157e93068dd) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.646569 341 ts_tablet_manager.cc:1905] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.647677 32139 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 3 candidate_status { last_received { term: 2 index: 263 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
I20250623 08:04:27.648566 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet d0ee5a4fe03848229524cc0105226d8c with delete_type TABLET_DATA_DELETED (replaced by tablet 38ecd4c243b840f4b66d8e01078dca64) from {username='slave'} at 127.0.0.1:55518
I20250623 08:04:27.650269 32353 tablet_service.cc:1468] Processing CreateTablet for tablet f7d69bcad8684d99a0b08eefb470ab01 (DEFAULT_TABLE table=default.Table_0bb2584ec3e0467194a9591bbe5ae477 [id=87e7893d85264d4c9cbcc5564f840390]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.650686 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d69bcad8684d99a0b08eefb470ab01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.652695 341 ts_tablet_manager.cc:1918] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.652815 341 log.cc:1199] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/e90151394fb842498f4e780acab1f1aa
I20250623 08:04:27.653155 341 ts_tablet_manager.cc:1939] T e90151394fb842498f4e780acab1f1aa P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.656630 343 tablet_bootstrap.cc:492] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.656800 343 ts_tablet_manager.cc:1397] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20250623 08:04:27.657516 343 raft_consensus.cc:357] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.657722 343 raft_consensus.cc:383] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.657800 343 raft_consensus.cc:738] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.658077 343 consensus_queue.cc:260] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.658592 343 ts_tablet_manager.cc:1428] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:27.658970 343 tablet_bootstrap.cc:492] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.659590 32493 tablet_service.cc:1468] Processing CreateTablet for tablet f7d69bcad8684d99a0b08eefb470ab01 (DEFAULT_TABLE table=default.Table_0bb2584ec3e0467194a9591bbe5ae477 [id=87e7893d85264d4c9cbcc5564f840390]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.660019 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d69bcad8684d99a0b08eefb470ab01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.660631 343 tablet_bootstrap.cc:654] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.660956 340 ts_tablet_manager.cc:1918] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:27.661089 340 log.cc:1199] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/7300f526d91f441bb164fe47f3a4f0ce
I20250623 08:04:27.661571 340 ts_tablet_manager.cc:1939] T 7300f526d91f441bb164fe47f3a4f0ce P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:27.661936 341 tablet_replica.cc:331] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.662104 341 raft_consensus.cc:2241] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.662267 341 raft_consensus.cc:2270] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.662784 341 ts_tablet_manager.cc:1905] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.667812 343 tablet_bootstrap.cc:492] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:27.667950 343 ts_tablet_manager.cc:1397] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250623 08:04:27.668680 343 raft_consensus.cc:357] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.668864 343 raft_consensus.cc:383] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.668954 343 raft_consensus.cc:738] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.669179 343 consensus_queue.cc:260] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.669826 343 ts_tablet_manager.cc:1428] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:27.670107 343 tablet_bootstrap.cc:492] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:27.669090 429 rpcz_store.cc:276] 0623 08:04:26.464285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.464341 (+ 56us) service_pool.cc:225] Handling call
0623 08:04:27.635179 (+1170838us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x616000070654. stack: 00007f917c618d56 00007f917c618b10 00007f917b60f29d 0000000000970a84 0000000000970a4d 00007f918b3b5fcd 00007f918b3b60c3 00007f918807a1c1 00007f918fbfa2ed 00007f918fd8ba15 00007f9186bdec17 00007f9186bde945 00007f91833bc93e 00007f91833bb81e 00007f91833c1a55 00007f91833c237f
0623 08:04:27.635836 (+ 657us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2689682304}
I20250623 08:04:27.671702 343 tablet_bootstrap.cc:654] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.673197 32620 tablet_service.cc:1468] Processing CreateTablet for tablet f7d69bcad8684d99a0b08eefb470ab01 (DEFAULT_TABLE table=default.Table_0bb2584ec3e0467194a9591bbe5ae477 [id=87e7893d85264d4c9cbcc5564f840390]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.673599 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d69bcad8684d99a0b08eefb470ab01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.674171 610 raft_consensus.cc:491] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.674351 610 raft_consensus.cc:513] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.675591 341 ts_tablet_manager.cc:1918] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.675707 341 log.cc:1199] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/a46c0a89fc014054a75ed198e824b5b6
I20250623 08:04:27.676074 341 ts_tablet_manager.cc:1939] T a46c0a89fc014054a75ed198e824b5b6 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.676936 610 leader_election.cc:290] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:27.679073 341 tablet_replica.cc:331] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.679262 341 raft_consensus.cc:2241] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.679435 341 raft_consensus.cc:2270] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.679936 341 ts_tablet_manager.cc:1905] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.679992 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c01121f2e644d6aa6e8ec228bf75be9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:27.680204 32520 raft_consensus.cc:2466] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:27.681252 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c01121f2e644d6aa6e8ec228bf75be9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.681425 32656 raft_consensus.cc:2466] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:27.681870 32314 leader_election.cc:304] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:27.682440 610 raft_consensus.cc:2802] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:27.682588 610 raft_consensus.cc:491] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:27.682718 610 raft_consensus.cc:3058] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.681021 560 leader_election.cc:290] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313)
I20250623 08:04:27.688482 610 raft_consensus.cc:513] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.689462 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c01121f2e644d6aa6e8ec228bf75be9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:27.689653 32656 raft_consensus.cc:3058] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:27.689702 432 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.1:59097 (request call id 17) took 1323 ms (client timeout 1000 ms). Trace:
I20250623 08:04:27.688426 391 leader_election.cc:304] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678; no voters: 9752155c20ee42fbb7ff7b7d5b682444, d31835ec4ad74332a00998d431492466
I20250623 08:04:27.691797 560 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250623 08:04:27.691179 432 rpcz_store.cc:269] 0623 08:04:26.362387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.364174 (+ 1787us) service_pool.cc:225] Handling call
0623 08:04:27.642966 (+1278792us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x616000070654. stack: 00007f917c618d56 00007f917c618b10 00007f917b60f29d 0000000000970a84 0000000000970a4d 00007f918b3b5fcd 00007f918b3b60c3 00007f918807a1c1 00007f918fbfa2ed 00007f918fd7db58 00007f9186bca176 00007f9186bc9ea5 00007f91833bc93e 00007f91833bb81e 00007f91833c1a55 00007f91833c237f
0623 08:04:27.685360 (+ 42394us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2933600512}
I20250623 08:04:27.692584 32656 raft_consensus.cc:2466] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:27.680903 635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:27.692938 635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:27.693424 32314 leader_election.cc:304] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:27.693718 641 raft_consensus.cc:2802] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
W20250623 08:04:27.697311 430 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:60995 (request call id 12) took 1317 ms (client timeout 1000 ms). Trace:
W20250623 08:04:27.697417 430 rpcz_store.cc:269] 0623 08:04:26.379421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.379482 (+ 61us) service_pool.cc:225] Handling call
0623 08:04:27.686111 (+1306629us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x616000070654. stack: 00007f917c618d56 00007f917c618b10 00007f917b60f29d 0000000000970a84 0000000000970a4d 00007f918b3b5fcd 00007f918b3b60c3 00007f918807a1c1 00007f918fbfa2ed 00007f918fd7db58 00007f9186bca176 00007f9186bc9ea5 00007f91833bc93e 00007f91833bb81e 00007f91833c1a55 00007f91833c237f
0623 08:04:27.697254 (+ 11143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2965812608}
W20250623 08:04:27.697955 431 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:58369 (request call id 11) took 1316 ms (client timeout 1000 ms). Trace:
W20250623 08:04:27.698051 431 rpcz_store.cc:269] 0623 08:04:26.375530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.375599 (+ 69us) service_pool.cc:225] Handling call
0623 08:04:27.679930 (+1304331us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x616000070654. stack: 00007f917c618d56 00007f917c618b10 00007f917b60f29d 0000000000970a84 0000000000970a4d 00007f918b3b5fcd 00007f918b3b60c3 00007f918807a1c1 00007f918fbfa2ed 00007f918fd7db58 00007f9186bca176 00007f9186bc9ea5 00007f91833bc93e 00007f91833bb81e 00007f91833c1a55 00007f91833c237f
0623 08:04:27.692435 (+ 12505us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2914020096}
I20250623 08:04:27.702066 610 leader_election.cc:290] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:27.702724 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c01121f2e644d6aa6e8ec228bf75be9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:27.702997 32520 raft_consensus.cc:3058] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.705857 32520 raft_consensus.cc:2466] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:27.715158 341 ts_tablet_manager.cc:1918] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.715339 341 log.cc:1199] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/c7385d86316f4a3ab68a77d18080ec20
I20250623 08:04:27.715730 341 ts_tablet_manager.cc:1939] T c7385d86316f4a3ab68a77d18080ec20 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.721005 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 (DEFAULT_TABLE table=default.Table_c2afde03c007450390a0466fe1d39344 [id=77e1411772b54c89a9a8435654249287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.721449 341 tablet_replica.cc:331] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.721637 341 raft_consensus.cc:2241] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.721776 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f9d21b8c144f9cada9bdbe0fbb1538. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.721814 341 raft_consensus.cc:2270] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.722373 341 ts_tablet_manager.cc:1905] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.728616 32348 tablet_service.cc:1468] Processing CreateTablet for tablet a414144276f6484ea4690204752e2142 (DEFAULT_TABLE table=default.Table_a4ca66fa08cc441ab5571f9968f30914 [id=c3c5e1869ba745b9b2ab29742806f241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.728953 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a414144276f6484ea4690204752e2142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.734638 457 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48180 (request call id 0) took 1303 ms. Trace:
I20250623 08:04:27.737008 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 (DEFAULT_TABLE table=default.Table_c2afde03c007450390a0466fe1d39344 [id=77e1411772b54c89a9a8435654249287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.737556 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f9d21b8c144f9cada9bdbe0fbb1538. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.742154 341 ts_tablet_manager.cc:1918] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.742362 341 log.cc:1199] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/04a3ae58d26142d58c4df4f52d7e3e2c
I20250623 08:04:27.742386 662 raft_consensus.cc:491] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.742741 341 ts_tablet_manager.cc:1939] T 04a3ae58d26142d58c4df4f52d7e3e2c P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.742723 662 raft_consensus.cc:513] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.744760 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a97b34e386443fa19a9601621b5017" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:27.745497 32453 leader_election.cc:343] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.745923 662 leader_election.cc:290] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:27.746321 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a97b34e386443fa19a9601621b5017" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:27.747051 32453 leader_election.cc:343] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.747167 32453 leader_election.cc:304] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.747730 662 raft_consensus.cc:2747] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.748251 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 (DEFAULT_TABLE table=default.Table_c2afde03c007450390a0466fe1d39344 [id=77e1411772b54c89a9a8435654249287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.748687 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f9d21b8c144f9cada9bdbe0fbb1538. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.755149 344 tablet_replica.cc:331] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:27.755633 344 raft_consensus.cc:2241] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.756461 344 raft_consensus.cc:2270] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.756493 341 tablet_replica.cc:331] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.756708 341 raft_consensus.cc:2241] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.756887 341 raft_consensus.cc:2270] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.757748 32630 tablet_service.cc:1468] Processing CreateTablet for tablet a414144276f6484ea4690204752e2142 (DEFAULT_TABLE table=default.Table_a4ca66fa08cc441ab5571f9968f30914 [id=c3c5e1869ba745b9b2ab29742806f241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.758226 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a414144276f6484ea4690204752e2142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.762696 341 ts_tablet_manager.cc:1905] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.765615 610 raft_consensus.cc:491] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.765924 610 raft_consensus.cc:513] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.766646 610 leader_election.cc:290] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:27.768240 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "228375592aee48a1aa21e1d16962b663" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.768512 32656 raft_consensus.cc:2466] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:27.769115 32314 leader_election.cc:304] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:27.769490 610 raft_consensus.cc:2802] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:27.769629 610 raft_consensus.cc:491] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:27.769771 610 raft_consensus.cc:3058] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.772889 610 raft_consensus.cc:513] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.776335 341 ts_tablet_manager.cc:1918] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.776295 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 (DEFAULT_TABLE table=default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c [id=387b3fc3d26b457099bcbb3b8ef3ac9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.776471 341 log.cc:1199] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/2eb99713f9214e6488d5c6e140ac263c
I20250623 08:04:27.776829 341 ts_tablet_manager.cc:1939] T 2eb99713f9214e6488d5c6e140ac263c P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.776969 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.784643 641 raft_consensus.cc:695] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:27.785044 641 consensus_queue.cc:237] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.787281 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 (DEFAULT_TABLE table=default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c [id=387b3fc3d26b457099bcbb3b8ef3ac9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.787706 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.797112 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "228375592aee48a1aa21e1d16962b663" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:27.797353 32656 raft_consensus.cc:3058] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.797717 341 tablet_replica.cc:331] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:27.797937 341 raft_consensus.cc:2241] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:27.798120 341 raft_consensus.cc:2270] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:27.798247 610 leader_election.cc:290] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:27.799439 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 0c58231f67604422a1497e4c1b44ec52 (DEFAULT_TABLE table=default.Table_fe81102858f244af9a4a286eaf6e34bf [id=8a0d9f5619d843148169561ffc7cfdc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.799579 341 ts_tablet_manager.cc:1905] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.799655 339 tablet_bootstrap.cc:492] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:27.799803 339 ts_tablet_manager.cc:1397] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.367s user 0.050s sys 0.082s
I20250623 08:04:27.799846 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c58231f67604422a1497e4c1b44ec52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.800844 339 raft_consensus.cc:357] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.801084 339 raft_consensus.cc:383] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.801179 339 raft_consensus.cc:738] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.801457 339 consensus_queue.cc:260] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.801985 339 ts_tablet_manager.cc:1428] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:27.802265 339 tablet_bootstrap.cc:492] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:27.809402 341 ts_tablet_manager.cc:1918] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:27.809523 341 log.cc:1199] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/69e772df590747ff841be5f218e55acf
I20250623 08:04:27.757402 32488 tablet_service.cc:1468] Processing CreateTablet for tablet a414144276f6484ea4690204752e2142 (DEFAULT_TABLE table=default.Table_a4ca66fa08cc441ab5571f9968f30914 [id=c3c5e1869ba745b9b2ab29742806f241]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.809702 339 tablet_bootstrap.cc:654] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.809859 341 ts_tablet_manager.cc:1939] T 69e772df590747ff841be5f218e55acf P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:27.810002 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a414144276f6484ea4690204752e2142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.767802 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "228375592aee48a1aa21e1d16962b663" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:27.810670 32520 raft_consensus.cc:2466] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:27.774247 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "228375592aee48a1aa21e1d16962b663" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:27.811189 32518 raft_consensus.cc:3058] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.782068 662 raft_consensus.cc:491] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.811962 662 raft_consensus.cc:513] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.812615 662 leader_election.cc:290] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:27.813445 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726d80e05a344f27b5a289723cb5aed7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:27.815758 32453 leader_election.cc:343] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.816356 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726d80e05a344f27b5a289723cb5aed7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:27.816896 32453 leader_election.cc:343] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:27.817080 32453 leader_election.cc:304] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.817499 662 raft_consensus.cc:2747] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.818300 344 ts_tablet_manager.cc:1905] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:27.823559 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 0c58231f67604422a1497e4c1b44ec52 (DEFAULT_TABLE table=default.Table_fe81102858f244af9a4a286eaf6e34bf [id=8a0d9f5619d843148169561ffc7cfdc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.823993 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c58231f67604422a1497e4c1b44ec52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.825167 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 0c58231f67604422a1497e4c1b44ec52 (DEFAULT_TABLE table=default.Table_fe81102858f244af9a4a286eaf6e34bf [id=8a0d9f5619d843148169561ffc7cfdc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.825552 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c58231f67604422a1497e4c1b44ec52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.825740 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (DEFAULT_TABLE table=default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.826051 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.827363 32623 tablet_service.cc:1468] Processing CreateTablet for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (DEFAULT_TABLE table=default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.827728 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.832190 32351 tablet_service.cc:1468] Processing CreateTablet for tablet b19e5f2359a84c7ea02a0b25240a54cb (DEFAULT_TABLE table=default.Table_640be163e2114433bd7079aa3a602116 [id=d422f6202d1842e998a23ec064757334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.832540 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b19e5f2359a84c7ea02a0b25240a54cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.826697 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (DEFAULT_TABLE table=default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.840520 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.849584 32518 raft_consensus.cc:2466] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:27.862561 32314 leader_election.cc:304] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:27.863119 610 raft_consensus.cc:2802] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:27.863304 610 raft_consensus.cc:695] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:27.863596 610 consensus_queue.cc:237] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.796002 32495 tablet_service.cc:1468] Processing CreateTablet for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 (DEFAULT_TABLE table=default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c [id=387b3fc3d26b457099bcbb3b8ef3ac9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.809274 657 consensus_queue.cc:1035] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:27.877513 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:27.887409 32453 consensus_peers.cc:487] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7300f526d91f441bb164fe47f3a4f0ce. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:27.903812 657 raft_consensus.cc:491] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.903997 657 raft_consensus.cc:513] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.904632 657 leader_election.cc:290] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:27.904866 32117 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4ac9f38877b24d619c280a1aaa0b7b08 (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4ac9f38877b24d619c280a1aaa0b7b08 consensus state has no leader
I20250623 08:04:27.906457 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16d6b1e81e4d438f85d8fee4dd7d8b7d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:27.906739 32653 raft_consensus.cc:2466] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:27.907161 32453 leader_election.cc:304] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:27.907490 657 raft_consensus.cc:2802] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:27.907609 657 raft_consensus.cc:491] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:27.907732 657 raft_consensus.cc:3058] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:27.911046 32453 leader_election.cc:343] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 16d6b1e81e4d438f85d8fee4dd7d8b7d
I20250623 08:04:27.912052 657 raft_consensus.cc:513] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:27.912879 657 leader_election.cc:290] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:27.913583 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16d6b1e81e4d438f85d8fee4dd7d8b7d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:27.913748 32653 raft_consensus.cc:3058] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:27.914084 32453 leader_election.cc:343] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 16d6b1e81e4d438f85d8fee4dd7d8b7d
I20250623 08:04:27.916422 32488 tablet_service.cc:1468] Processing CreateTablet for tablet b19e5f2359a84c7ea02a0b25240a54cb (DEFAULT_TABLE table=default.Table_640be163e2114433bd7079aa3a602116 [id=d422f6202d1842e998a23ec064757334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.916967 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b19e5f2359a84c7ea02a0b25240a54cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.924427 657 consensus_queue.cc:1035] T c1596e80225041099fb513ce477eae33 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:27.931800 32627 tablet_service.cc:1468] Processing CreateTablet for tablet b19e5f2359a84c7ea02a0b25240a54cb (DEFAULT_TABLE table=default.Table_640be163e2114433bd7079aa3a602116 [id=d422f6202d1842e998a23ec064757334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.932329 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b19e5f2359a84c7ea02a0b25240a54cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.950302 32351 tablet_service.cc:1468] Processing CreateTablet for tablet eef663ffdaf141aea5e6269b6b2e1c89 (DEFAULT_TABLE table=default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.950960 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef663ffdaf141aea5e6269b6b2e1c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.959988 342 tablet_bootstrap.cc:492] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:27.960191 342 ts_tablet_manager.cc:1397] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.384s user 0.021s sys 0.091s
I20250623 08:04:27.961349 342 raft_consensus.cc:357] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.961586 342 raft_consensus.cc:383] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:27.961719 342 raft_consensus.cc:738] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:27.961995 342 consensus_queue.cc:260] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:27.962711 342 ts_tablet_manager.cc:1428] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:27.964092 342 tablet_bootstrap.cc:492] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:27.966720 342 tablet_bootstrap.cc:654] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:27.972390 32491 tablet_service.cc:1468] Processing CreateTablet for tablet eef663ffdaf141aea5e6269b6b2e1c89 (DEFAULT_TABLE table=default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.972851 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef663ffdaf141aea5e6269b6b2e1c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:27.979732 587 raft_consensus.cc:491] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:27.979945 587 raft_consensus.cc:513] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:27.980767 587 leader_election.cc:290] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:27.981788 32589 leader_election.cc:343] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 7fea0b5aa0c1403fb3ef3ab9ab040035
W20250623 08:04:27.989442 32589 leader_election.cc:343] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 7fea0b5aa0c1403fb3ef3ab9ab040035
I20250623 08:04:27.989586 32589 leader_election.cc:304] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:27.990170 587 raft_consensus.cc:2747] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:27.990921 32633 tablet_service.cc:1468] Processing CreateTablet for tablet eef663ffdaf141aea5e6269b6b2e1c89 (DEFAULT_TABLE table=default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:27.991418 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef663ffdaf141aea5e6269b6b2e1c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.000631 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 1a070895f4984493a3b9f6fa779797a9 (DEFAULT_TABLE table=default.Table_79f69fca0b25496e896d93fdd78e14b6 [id=cf095cc479d741b8b7c2cd352724713a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.001160 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a070895f4984493a3b9f6fa779797a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.003526 342 tablet_bootstrap.cc:492] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.003728 342 ts_tablet_manager.cc:1397] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.003s
I20250623 08:04:28.004721 342 raft_consensus.cc:357] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.004963 342 raft_consensus.cc:383] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.005056 342 raft_consensus.cc:738] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.005321 342 consensus_queue.cc:260] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.005755 623 mvcc.cc:204] Tried to move back new op lower bound from 7170727386157494272 to 7170727371776569344. Current Snapshot: MvccSnapshot[applied={T|T < 7170727386157494272 or (T in {7170727386157494272})}]
I20250623 08:04:28.010433 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 1a070895f4984493a3b9f6fa779797a9 (DEFAULT_TABLE table=default.Table_79f69fca0b25496e896d93fdd78e14b6 [id=cf095cc479d741b8b7c2cd352724713a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.010926 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a070895f4984493a3b9f6fa779797a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.016600 698 tablet.cc:1722] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:28.026908 701 tablet.cc:1722] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:28.028362 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]) successfully deleted
W20250623 08:04:28.028724 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 2060a643d0da47efa868f52bdb331b1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2060a643d0da47efa868f52bdb331b1f already in progress: creating tablet
W20250623 08:04:28.028991 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 279624b8bb834b95a4f08045968ebfb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 279624b8bb834b95a4f08045968ebfb4 already in progress: creating tablet
W20250623 08:04:28.029196 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet ae7c2ac546114a819a543f1667fb7844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae7c2ac546114a819a543f1667fb7844 already in progress: creating tablet
W20250623 08:04:28.029562 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 417a050eada4403ebbddff255fb6fb5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417a050eada4403ebbddff255fb6fb5a already in progress: creating tablet
W20250623 08:04:28.029793 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet b980c746a3c042bfae90aa076acbb1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b980c746a3c042bfae90aa076acbb1df already in progress: creating tablet
W20250623 08:04:28.030017 32114 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 79731106915d4cebb9dcc3c10386ffa9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79731106915d4cebb9dcc3c10386ffa9 already in progress: creating tablet
I20250623 08:04:28.030578 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet ce1f3ef3690c4837b95b105733884f1f (table default.Table_5e8e88b2237f435595d619af47384b4e [id=dec4e6cfe22942698b2f2ae716f72e92]) successfully deleted
I20250623 08:04:28.031812 341 tablet_replica.cc:331] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:28.032043 341 raft_consensus.cc:2241] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:28.032195 341 raft_consensus.cc:2270] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:28.032752 341 ts_tablet_manager.cc:1905] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:28.037485 700 raft_consensus.cc:491] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9752155c20ee42fbb7ff7b7d5b682444)
I20250623 08:04:28.039270 700 raft_consensus.cc:513] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:28.039709 341 ts_tablet_manager.cc:1918] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:28.039839 341 log.cc:1199] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/7fea0b5aa0c1403fb3ef3ab9ab040035
I20250623 08:04:28.040199 341 ts_tablet_manager.cc:1939] T 7fea0b5aa0c1403fb3ef3ab9ab040035 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:28.040566 32656 raft_consensus.cc:2466] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.041918 342 ts_tablet_manager.cc:1428] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.038s user 0.000s sys 0.008s
I20250623 08:04:28.043386 32653 raft_consensus.cc:2466] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:28.044920 32453 leader_election.cc:304] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.045717 699 raft_consensus.cc:2802] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.045933 699 raft_consensus.cc:695] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:28.046303 699 consensus_queue.cc:237] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.050217 342 tablet_bootstrap.cc:492] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.052337 342 tablet_bootstrap.cc:654] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.056113 32139 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31835ec4ad74332a00998d431492466" candidate_term: 3 candidate_status { last_received { term: 2 index: 347 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
I20250623 08:04:28.058439 341 tablet_replica.cc:331] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:28.063520 700 leader_election.cc:290] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:28.064440 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 1a070895f4984493a3b9f6fa779797a9 (DEFAULT_TABLE table=default.Table_79f69fca0b25496e896d93fdd78e14b6 [id=cf095cc479d741b8b7c2cd352724713a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.065136 32359 tablet_service.cc:1468] Processing CreateTablet for tablet d98654c29b9c4e75af6a213cafd4c2c9 (DEFAULT_TABLE table=default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.068197 341 raft_consensus.cc:2241] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:28.071408 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 9c4dec1a48f84fb3bd7bbd757f7b0c1b (table default.Table_0b402e459582438db7f3ad072b276ee0 [id=562e3d3edc3f4341ae8909d9fcf2e6c2]) successfully deleted
I20250623 08:04:28.071666 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet d67a8f1f570741ebb886c4d136ca40d2 (table default.Table_be622804eb114901b245383926c3afec [id=b020e49a601748e9bcbf397e235dc596]) successfully deleted
I20250623 08:04:28.071885 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 35428260e05e447ebf88ea41e63cdd02 (table default.Table_c5dbc5b105b8416c90d3792d29568b83 [id=a1839631f0c64b16a3f0411b514f93a2]) successfully deleted
I20250623 08:04:28.071872 685 consensus_queue.cc:1035] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 08:04:28.072055 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet e90151394fb842498f4e780acab1f1aa (table default.Table_eefa8d393087436793f03c22b76ab152 [id=a1d4326f22fa44ba82cb376970f55ddc]) successfully deleted
W20250623 08:04:28.072280 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet ea702abb31fb41498ab61bbb05b1f7e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea702abb31fb41498ab61bbb05b1f7e2 already in progress: creating tablet
I20250623 08:04:28.072490 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet a46c0a89fc014054a75ed198e824b5b6 (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]) successfully deleted
W20250623 08:04:28.072722 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 2060a643d0da47efa868f52bdb331b1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2060a643d0da47efa868f52bdb331b1f already in progress: creating tablet
I20250623 08:04:28.073016 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet c7385d86316f4a3ab68a77d18080ec20 (table default.Table_465c441ac95145f1b498665ccedafe6c [id=24b0c7dcb97e4105b8edf674a0a90e84]) successfully deleted
W20250623 08:04:28.073220 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet ae7c2ac546114a819a543f1667fb7844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae7c2ac546114a819a543f1667fb7844 already in progress: creating tablet
W20250623 08:04:28.073442 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 279624b8bb834b95a4f08045968ebfb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 279624b8bb834b95a4f08045968ebfb4 already in progress: creating tablet
W20250623 08:04:28.073642 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 417a050eada4403ebbddff255fb6fb5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417a050eada4403ebbddff255fb6fb5a already in progress: creating tablet
W20250623 08:04:28.073884 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b980c746a3c042bfae90aa076acbb1df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b980c746a3c042bfae90aa076acbb1df already in progress: creating tablet
I20250623 08:04:28.074097 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 04a3ae58d26142d58c4df4f52d7e3e2c (table default.Table_fe1953ea80584690bccd51204d81a314 [id=7c2dc728796445d89dce65fce8b36f14]) successfully deleted
W20250623 08:04:28.074321 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 79731106915d4cebb9dcc3c10386ffa9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79731106915d4cebb9dcc3c10386ffa9 already in progress: creating tablet
I20250623 08:04:28.074659 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 2eb99713f9214e6488d5c6e140ac263c (table default.Table_2807052f51c24614b4068e8113db2abd [id=9e90ae34db704c408759a60271e0bf82]) successfully deleted
I20250623 08:04:28.074851 587 raft_consensus.cc:491] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.075496 343 tablet_bootstrap.cc:492] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.075609 342 tablet_bootstrap.cc:492] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.075754 342 ts_tablet_manager.cc:1397] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.003s
I20250623 08:04:28.076458 342 raft_consensus.cc:357] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.076632 342 raft_consensus.cc:383] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.076732 342 raft_consensus.cc:738] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.078328 587 raft_consensus.cc:513] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.079049 587 leader_election.cc:290] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.079725 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "696f5d2797ef463bb72a2119501c6ef3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.079916 32520 raft_consensus.cc:2466] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:28.080343 32589 leader_election.cc:304] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:28.080730 587 raft_consensus.cc:2802] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.080818 587 raft_consensus.cc:491] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.080906 587 raft_consensus.cc:3058] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.082139 343 ts_tablet_manager.cc:1397] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.406s user 0.046s sys 0.117s
I20250623 08:04:28.083230 343 raft_consensus.cc:357] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.083503 344 ts_tablet_manager.cc:1918] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:28.083581 343 raft_consensus.cc:383] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.083659 344 log.cc:1199] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/04eda92e14144733ab71a577dd3bd2b0
I20250623 08:04:28.083675 343 raft_consensus.cc:738] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.083967 343 consensus_queue.cc:260] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.084070 344 ts_tablet_manager.cc:1939] T 04eda92e14144733ab71a577dd3bd2b0 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:28.085033 344 tablet_replica.cc:331] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:28.085222 344 raft_consensus.cc:2241] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:28.085414 344 raft_consensus.cc:2270] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:28.085659 342 consensus_queue.cc:260] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:28.091372 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 966a159e212040109e2842c15b69fb71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 966a159e212040109e2842c15b69fb71 already in progress: creating tablet
W20250623 08:04:28.091643 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet e22f9bae5f7248af99cec06ce6dbbd41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e22f9bae5f7248af99cec06ce6dbbd41 already in progress: creating tablet
W20250623 08:04:28.091831 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 175ee90133994154b7e4b58b0770db3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 175ee90133994154b7e4b58b0770db3b already in progress: creating tablet
W20250623 08:04:28.092037 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 4cb66c7a273a412db3ae77af9e30d036 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cb66c7a273a412db3ae77af9e30d036 already in progress: creating tablet
W20250623 08:04:28.092221 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 8dc696236a15420e9666cca708068fb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dc696236a15420e9666cca708068fb2 already in progress: creating tablet
W20250623 08:04:28.092412 32114 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 205da880685b49b3b8669f73dc2c563b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 205da880685b49b3b8669f73dc2c563b already in progress: creating tablet
I20250623 08:04:28.092566 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 69e772df590747ff841be5f218e55acf (table default.Table_c3e871b94f3944d6b2b75626c3af94ec [id=a61a71f32ae444338b041f6e0a9fa286]) successfully deleted
I20250623 08:04:28.093225 32114 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 7fea0b5aa0c1403fb3ef3ab9ab040035 (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]) successfully deleted
I20250623 08:04:28.092464 342 ts_tablet_manager.cc:1428] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250623 08:04:28.093720 342 tablet_bootstrap.cc:492] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.094097 32489 tablet_service.cc:1468] Processing CreateTablet for tablet d98654c29b9c4e75af6a213cafd4c2c9 (DEFAULT_TABLE table=default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.094645 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d98654c29b9c4e75af6a213cafd4c2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.097517 339 tablet_bootstrap.cc:492] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.097700 342 tablet_bootstrap.cc:654] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.097715 339 ts_tablet_manager.cc:1397] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.296s user 0.030s sys 0.031s
I20250623 08:04:28.098270 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a070895f4984493a3b9f6fa779797a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.098578 339 raft_consensus.cc:357] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.098785 339 raft_consensus.cc:383] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.098856 339 raft_consensus.cc:738] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.098909 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d98654c29b9c4e75af6a213cafd4c2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.099095 339 consensus_queue.cc:260] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.099928 339 ts_tablet_manager.cc:1428] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:28.100200 699 raft_consensus.cc:491] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.100255 339 tablet_bootstrap.cc:492] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.100358 699 raft_consensus.cc:513] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.101415 699 leader_election.cc:290] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.101994 339 tablet_bootstrap.cc:654] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.104189 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0311ce67989f49e8866234a88e5b4d09" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:28.107052 32453 leader_election.cc:343] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.111773 342 tablet_bootstrap.cc:492] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.111928 342 ts_tablet_manager.cc:1397] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
W20250623 08:04:28.112268 32589 leader_election.cc:343] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 696f5d2797ef463bb72a2119501c6ef3
I20250623 08:04:28.112715 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0311ce67989f49e8866234a88e5b4d09" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.112732 342 raft_consensus.cc:357] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.112895 342 raft_consensus.cc:383] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.112962 342 raft_consensus.cc:738] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.113194 342 consensus_queue.cc:260] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:28.113428 32453 leader_election.cc:343] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.113528 32453 leader_election.cc:304] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.113817 342 ts_tablet_manager.cc:1428] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250623 08:04:28.114132 342 tablet_bootstrap.cc:492] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.114076 456 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31835ec4ad74332a00998d431492466" candidate_term: 3 candidate_status { last_received { term: 2 index: 347 } } ignore_live_leader: false dest_uuid: "6644af0f4b2a4b998282b021ec1dc678" is_pre_election: true
I20250623 08:04:28.114347 699 raft_consensus.cc:2747] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.114584 456 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d31835ec4ad74332a00998d431492466 in term 2.
I20250623 08:04:28.115216 32043 leader_election.cc:304] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678, d31835ec4ad74332a00998d431492466; no voters: 9752155c20ee42fbb7ff7b7d5b682444
I20250623 08:04:28.085906 344 ts_tablet_manager.cc:1905] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:28.117543 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 (DEFAULT_TABLE table=default.Table_e7f8192c00524b5899152fa3f717b898 [id=f4277b270809454da8cfd8f3158ed6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.117678 342 tablet_bootstrap.cc:654] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.118043 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.119567 339 tablet_bootstrap.cc:492] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.119745 339 ts_tablet_manager.cc:1397] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250623 08:04:28.120165 587 raft_consensus.cc:513] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.120759 339 raft_consensus.cc:357] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.120958 339 raft_consensus.cc:383] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.121070 339 raft_consensus.cc:738] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.121343 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "696f5d2797ef463bb72a2119501c6ef3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.121572 32520 raft_consensus.cc:3058] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.121359 339 consensus_queue.cc:260] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.122313 339 ts_tablet_manager.cc:1428] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:28.122623 339 tablet_bootstrap.cc:492] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.124527 32520 raft_consensus.cc:2466] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:28.124621 339 tablet_bootstrap.cc:654] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.125008 32589 leader_election.cc:304] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:28.125370 32488 tablet_service.cc:1468] Processing CreateTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 (DEFAULT_TABLE table=default.Table_e7f8192c00524b5899152fa3f717b898 [id=f4277b270809454da8cfd8f3158ed6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.125768 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.128460 714 raft_consensus.cc:491] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.128676 714 raft_consensus.cc:513] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:28.129220 32589 leader_election.cc:343] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 696f5d2797ef463bb72a2119501c6ef3
I20250623 08:04:28.130307 32361 tablet_service.cc:1468] Processing CreateTablet for tablet b80a91ae2c084020b5d3444b570b28f2 (DEFAULT_TABLE table=default.Table_d226f9dab24f45ae896579983b3d1a67 [id=e0cd5a7f63b443c2a3e86ec75a1e3778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.130376 587 leader_election.cc:290] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.130685 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b80a91ae2c084020b5d3444b570b28f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:28.130954 32589 leader_election.cc:343] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 04eda92e14144733ab71a577dd3bd2b0
I20250623 08:04:28.131135 587 raft_consensus.cc:2802] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.131336 587 raft_consensus.cc:695] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:28.131639 587 consensus_queue.cc:237] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.136574 341 raft_consensus.cc:2270] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:28.138700 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (DEFAULT_TABLE table=default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.139217 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.143340 341 ts_tablet_manager.cc:1905] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:28.146548 32114 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 7300f526d91f441bb164fe47f3a4f0ce (table default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517]) successfully deleted
W20250623 08:04:28.147380 32589 leader_election.cc:343] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 04eda92e14144733ab71a577dd3bd2b0
I20250623 08:04:28.147439 32491 tablet_service.cc:1468] Processing CreateTablet for tablet b80a91ae2c084020b5d3444b570b28f2 (DEFAULT_TABLE table=default.Table_d226f9dab24f45ae896579983b3d1a67 [id=e0cd5a7f63b443c2a3e86ec75a1e3778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.147541 32589 leader_election.cc:304] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.148016 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b80a91ae2c084020b5d3444b570b28f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.148897 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (DEFAULT_TABLE table=default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.150174 339 tablet_bootstrap.cc:492] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.150331 339 ts_tablet_manager.cc:1397] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.005s
I20250623 08:04:28.150898 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.151247 339 raft_consensus.cc:357] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.151440 339 raft_consensus.cc:383] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.151515 339 raft_consensus.cc:738] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.151786 339 consensus_queue.cc:260] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.152455 339 ts_tablet_manager.cc:1428] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:28.152928 339 tablet_bootstrap.cc:492] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.153301 32114 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 04eda92e14144733ab71a577dd3bd2b0 (table default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a [id=89a3fc1cb42f4902a69ea52945cedc0c]) successfully deleted
I20250623 08:04:28.157940 339 tablet_bootstrap.cc:654] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.166152 339 tablet_bootstrap.cc:492] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.166301 339 ts_tablet_manager.cc:1397] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.002s
I20250623 08:04:28.167248 339 raft_consensus.cc:357] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.167450 339 raft_consensus.cc:383] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.167548 339 raft_consensus.cc:738] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.169760 339 consensus_queue.cc:260] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.170763 339 ts_tablet_manager.cc:1428] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:28.171058 339 tablet_bootstrap.cc:492] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.174836 697 raft_consensus.cc:491] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.175101 697 raft_consensus.cc:513] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.176389 697 leader_election.cc:290] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.176781 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f02f36d6c1174a23a1f1043a431d0b6b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.177346 339 tablet_bootstrap.cc:654] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.177395 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f02f36d6c1174a23a1f1043a431d0b6b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.177579 32381 raft_consensus.cc:2466] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:28.178495 32453 leader_election.cc:343] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.179093 32453 leader_election.cc:304] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.179510 697 raft_consensus.cc:2802] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.179984 697 raft_consensus.cc:491] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.180104 697 raft_consensus.cc:3058] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.181617 709 raft_consensus.cc:2747] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.183044 32620 tablet_service.cc:1468] Processing CreateTablet for tablet d98654c29b9c4e75af6a213cafd4c2c9 (DEFAULT_TABLE table=default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.183522 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d98654c29b9c4e75af6a213cafd4c2c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.183619 339 tablet_bootstrap.cc:492] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.183770 339 ts_tablet_manager.cc:1397] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250623 08:04:28.184473 339 raft_consensus.cc:357] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.184659 339 raft_consensus.cc:383] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.184738 339 raft_consensus.cc:738] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.184947 339 consensus_queue.cc:260] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.187199 339 ts_tablet_manager.cc:1428] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:28.187476 339 tablet_bootstrap.cc:492] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.189013 339 tablet_bootstrap.cc:654] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.194226 339 tablet_bootstrap.cc:492] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.194351 339 ts_tablet_manager.cc:1397] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250623 08:04:28.195041 339 raft_consensus.cc:357] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.195211 339 raft_consensus.cc:383] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.195284 339 raft_consensus.cc:738] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.196970 339 consensus_queue.cc:260] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.197732 343 ts_tablet_manager.cc:1428] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.115s user 0.005s sys 0.001s
I20250623 08:04:28.197930 339 ts_tablet_manager.cc:1428] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:28.198218 339 tablet_bootstrap.cc:492] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.202503 339 tablet_bootstrap.cc:654] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.202900 699 raft_consensus.cc:491] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.204406 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 (DEFAULT_TABLE table=default.Table_e7f8192c00524b5899152fa3f717b898 [id=f4277b270809454da8cfd8f3158ed6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.204612 343 tablet_bootstrap.cc:492] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.204824 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.206401 343 tablet_bootstrap.cc:654] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.207056 32633 tablet_service.cc:1468] Processing CreateTablet for tablet b80a91ae2c084020b5d3444b570b28f2 (DEFAULT_TABLE table=default.Table_d226f9dab24f45ae896579983b3d1a67 [id=e0cd5a7f63b443c2a3e86ec75a1e3778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.207512 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b80a91ae2c084020b5d3444b570b28f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.209352 714 leader_election.cc:290] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.211040 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (DEFAULT_TABLE table=default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:28.211401 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:28.218014 610 raft_consensus.cc:491] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.218204 610 raft_consensus.cc:513] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.219151 610 leader_election.cc:290] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.220023 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:28.220603 32314 leader_election.cc:343] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.220862 699 raft_consensus.cc:513] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.225311 339 tablet_bootstrap.cc:492] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.225479 339 ts_tablet_manager.cc:1397] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.011s
I20250623 08:04:28.226445 32020 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskJxinPB/build/asan/bin/kudu pid 32096
I20250623 08:04:28.226545 662 consensus_queue.cc:1035] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:28.227433 339 raft_consensus.cc:357] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.227672 339 raft_consensus.cc:383] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.227766 339 raft_consensus.cc:738] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.228011 339 consensus_queue.cc:260] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.228534 339 ts_tablet_manager.cc:1428] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250623 08:04:28.228633 32160 generic_service.cc:196] Checking for leaks (request via RPC)
I20250623 08:04:28.228823 339 tablet_bootstrap.cc:492] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.231001 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.231200 32379 raft_consensus.cc:2466] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:28.231604 610 raft_consensus.cc:491] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.231782 610 raft_consensus.cc:513] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.232350 610 leader_election.cc:290] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.233268 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe30079045d437aa61bb5bfbfb96133" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.233347 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.233593 32453 leader_election.cc:304] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.233930 342 tablet_bootstrap.cc:492] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.233897 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe30079045d437aa61bb5bfbfb96133" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.234059 342 ts_tablet_manager.cc:1397] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.120s user 0.020s sys 0.019s
I20250623 08:04:28.234954 699 leader_election.cc:290] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.234968 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.235167 339 tablet_bootstrap.cc:654] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:28.235469 32314 leader_election.cc:343] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:28.235751 32314 leader_election.cc:343] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.235893 32314 leader_election.cc:304] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
W20250623 08:04:28.236404 32453 leader_election.cc:343] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.236423 662 raft_consensus.cc:491] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.236869 662 raft_consensus.cc:513] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.238023 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "776b412b3f564a7581c6a5e1c84e8499" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.238230 32379 raft_consensus.cc:2466] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:28.238231 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "776b412b3f564a7581c6a5e1c84e8499" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.238646 610 raft_consensus.cc:2747] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.239586 32520 raft_consensus.cc:2466] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.240391 700 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250623 08:04:28.240801 700 raft_consensus.cc:491] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9752155c20ee42fbb7ff7b7d5b682444)
I20250623 08:04:28.240921 700 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 2 FOLLOWER]: Advancing to term 3
I20250623 08:04:28.241271 32453 leader_election.cc:304] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:28.241529 32453 leader_election.cc:343] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.241978 662 leader_election.cc:290] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.245224 32314 leader_election.cc:304] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.242056 699 raft_consensus.cc:2802] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.245667 342 raft_consensus.cc:357] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.245823 342 raft_consensus.cc:383] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.246104 342 raft_consensus.cc:738] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.246328 610 raft_consensus.cc:2802] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.246333 342 consensus_queue.cc:260] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.246440 610 raft_consensus.cc:491] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.246526 610 raft_consensus.cc:3058] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.249562 610 raft_consensus.cc:513] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.251937 700 raft_consensus.cc:513] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:28.252036 662 raft_consensus.cc:2802] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.253353 32139 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31835ec4ad74332a00998d431492466" candidate_term: 3 candidate_status { last_received { term: 2 index: 356 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444"
I20250623 08:04:28.253661 456 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d31835ec4ad74332a00998d431492466" candidate_term: 3 candidate_status { last_received { term: 2 index: 356 } } ignore_live_leader: false dest_uuid: "6644af0f4b2a4b998282b021ec1dc678"
I20250623 08:04:28.253919 456 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 2 FOLLOWER]: Advancing to term 3
I20250623 08:04:28.252967 700 leader_election.cc:290] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 3 election: Requested vote from peers 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:28.254772 699 raft_consensus.cc:491] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.255326 699 raft_consensus.cc:3058] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.255748 342 ts_tablet_manager.cc:1428] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250623 08:04:28.255688 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.257344 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
W20250623 08:04:28.258028 32314 leader_election.cc:343] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.258316 610 leader_election.cc:290] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.258244 456 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d31835ec4ad74332a00998d431492466 in term 3.
I20250623 08:04:28.259249 32043 leader_election.cc:304] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678, d31835ec4ad74332a00998d431492466; no voters: 9752155c20ee42fbb7ff7b7d5b682444
I20250623 08:04:28.260126 700 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 3 FOLLOWER]: Leader election won for term 3
I20250623 08:04:28.261819 342 tablet_bootstrap.cc:492] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.263144 697 raft_consensus.cc:513] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.263899 697 leader_election.cc:290] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.264130 662 raft_consensus.cc:491] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.264245 662 raft_consensus.cc:3058] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.264459 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f02f36d6c1174a23a1f1043a431d0b6b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.265328 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f02f36d6c1174a23a1f1043a431d0b6b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.265532 32379 raft_consensus.cc:3058] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:27.734790 457 rpcz_store.cc:276] 0623 08:04:26.341767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:26.341828 (+ 61us) service_pool.cc:225] Handling call
0623 08:04:26.349304 (+ 7476us) raft_consensus.cc:1469] Updating replica for 238 ops
0623 08:04:26.627042 (+277738us) raft_consensus.cc:1532] Early marking committed up to index 25
0623 08:04:26.680438 (+ 53396us) raft_consensus.cc:1537] Triggering prepare for 238 ops
0623 08:04:26.680948 (+ 510us) write_op.cc:270] Start()
0623 08:04:26.680995 (+ 47us) write_op.cc:276] Timestamp: P: 1750665862803122 usec, L: 0
0623 08:04:26.684986 (+ 3991us) write_op.cc:270] Start()
0623 08:04:26.685032 (+ 46us) write_op.cc:276] Timestamp: P: 1750665862804040 usec, L: 0
0623 08:04:26.685220 (+ 188us) write_op.cc:270] Start()
0623 08:04:26.685240 (+ 20us) write_op.cc:276] Timestamp: P: 1750665862806577 usec, L: 0
0623 08:04:26.685365 (+ 125us) write_op.cc:270] Start()
0623 08:04:26.685384 (+ 19us) write_op.cc:276] Timestamp: P: 1750665862807294 usec, L: 0
0623 08:04:26.685521 (+ 137us) write_op.cc:270] Start()
0623 08:04:26.685540 (+ 19us) write_op.cc:276] Timestamp: P: 1750665862807887 usec, L: 0
0623 08:04:26.685686 (+ 146us) write_op.cc:270] Start()
0623 08:04:26.685709 (+ 23us) write_op.cc:276] Timestamp: P: 1750665862808871 usec, L: 0
0623 08:04:26.685843 (+ 134us) write_op.cc:270] Start()
0623 08:04:26.685884 (+ 41us) write_op.cc:276] Timestamp: P: 1750665862810015 usec, L: 0
0623 08:04:26.686009 (+ 125us) write_op.cc:270] Start()
0623 08:04:26.686030 (+ 21us) write_op.cc:276] Timestamp: P: 1750665862810554 usec, L: 0
0623 08:04:26.686151 (+ 121us) write_op.cc:270] Start()
0623 08:04:26.686172 (+ 21us) write_op.cc:276] Timestamp: P: 1750665862811118 usec, L: 0
0623 08:04:26.686344 (+ 172us) write_op.cc:270] Start()
0623 08:04:26.686365 (+ 21us) write_op.cc:276] Timestamp: P: 1750665862811716 usec, L: 0
0623 08:04:26.686482 (+ 117us) write_op.cc:270] Start()
0623 08:04:26.686512 (+ 30us) write_op.cc:276] Timestamp: P: 1750665862815438 usec, L: 0
0623 08:04:26.686669 (+ 157us) write_op.cc:270] Start()
0623 08:04:26.686690 (+ 21us) write_op.cc:276] Timestamp: P: 1750665862816623 usec, L: 0
0623 08:04:26.689097 (+ 2407us) write_op.cc:270] Start()
0623 08:04:26.689122 (+ 25us) write_op.cc:276] Timestamp: P: 1750665862829441 usec, L: 0
0623 08:04:26.689262 (+ 140us) write_op.cc:270] Start()
0623 08:04:26.689277 (+ 15us) write_op.cc:276] Timestamp: P: 1750665862830116 usec, L: 0
0623 08:04:26.691869 (+ 2592us) write_op.cc:270] Start()
0623 08:04:26.691896 (+ 27us) write_op.cc:276] Timestamp: P: 1750665862830851 usec, L: 0
0623 08:04:26.692044 (+ 148us) write_op.cc:270] Start()
0623 08:04:26.692079 (+ 35us) write_op.cc:276] Timestamp: P: 1750665862831803 usec, L: 0
0623 08:04:26.692218 (+ 139us) write_op.cc:270] Start()
0623 08:04:26.692237 (+ 19us) write_op.cc:276] Timestamp: P: 1750665862851859 usec, L: 0
0623 08:04:26.692387 (+ 150us) write_op.cc:270] Start()
0623 08:04:26.692408 (+ 21us) write_op.cc:276] Timestamp: P: 1750665862852572 usec, L: 0
0623 08:04:26.692540 (+ 132us) write_op.cc:270] Start()
0623 08:04:26.692559 (+ 19us) write_op.cc:276] Timestamp: P: 1750665862864467 usec, L: 0
0623 08:04:26.692717 (+ 158us) write_op.cc:270] Start()
0623 08:04:26.692737 (+ 20us) write_op.cc:276] Timestamp: P: 1750665862865233 usec, L: 0
0623 08:04:26.692850 (+ 113us) write_op.cc:270] Start()
0623 08:04:26.692867 (+ 17us) write_op.cc:276] Timestamp: P: 1750665862865953 usec, L: 0
0623 08:04:26.692971 (+ 104us) write_op.cc:270] Start()
0623 08:04:26.692988 (+ 17us) write_op.cc:276] Timestamp: P: 1750665862945904 usec, L: 0
0623 08:04:26.693135 (+ 147us) write_op.cc:270] Start()
0623 08:04:26.693154 (+ 19us) write_op.cc:276] Timestamp: P: 1750665863003749 usec, L: 0
0623 08:04:26.693267 (+ 113us) write_op.cc:270] Start()
0623 08:04:26.693284 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863026829 usec, L: 0
0623 08:04:26.693481 (+ 197us) write_op.cc:270] Start()
0623 08:04:26.693504 (+ 23us) write_op.cc:276] Timestamp: P: 1750665863027954 usec, L: 0
0623 08:04:26.693640 (+ 136us) write_op.cc:270] Start()
0623 08:04:26.693659 (+ 19us) write_op.cc:276] Timestamp: P: 1750665863029759 usec, L: 0
0623 08:04:26.693815 (+ 156us) write_op.cc:270] Start()
0623 08:04:26.693834 (+ 19us) write_op.cc:276] Timestamp: P: 1750665863043551 usec, L: 0
0623 08:04:26.693959 (+ 125us) write_op.cc:270] Start()
0623 08:04:26.693977 (+ 18us) write_op.cc:276] Timestamp: P: 1750665863044186 usec, L: 0
0623 08:04:26.694090 (+ 113us) write_op.cc:270] Start()
0623 08:04:26.694107 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863059393 usec, L: 0
0623 08:04:26.694266 (+ 159us) write_op.cc:270] Start()
0623 08:04:26.694289 (+ 23us) write_op.cc:276] Timestamp: P: 1750665863060047 usec, L: 0
0623 08:04:26.694424 (+ 135us) write_op.cc:270] Start()
0623 08:04:26.694443 (+ 19us) write_op.cc:276] Timestamp: P: 1750665863060611 usec, L: 0
0623 08:04:26.694555 (+ 112us) write_op.cc:270] Start()
0623 08:04:26.694573 (+ 18us) write_op.cc:276] Timestamp: P: 1750665863073781 usec, L: 0
0623 08:04:26.694706 (+ 133us) write_op.cc:270] Start()
0623 08:04:26.694726 (+ 20us) write_op.cc:276] Timestamp: P: 1750665863091051 usec, L: 0
0623 08:04:26.724287 (+ 29561us) write_op.cc:270] Start()
0623 08:04:26.724320 (+ 33us) write_op.cc:276] Timestamp: P: 1750665863093374 usec, L: 0
0623 08:04:26.724546 (+ 226us) write_op.cc:270] Start()
0623 08:04:26.724574 (+ 28us) write_op.cc:276] Timestamp: P: 1750665863096519 usec, L: 0
0623 08:04:26.724727 (+ 153us) write_op.cc:270] Start()
0623 08:04:26.724751 (+ 24us) write_op.cc:276] Timestamp: P: 1750665863115329 usec, L: 0
0623 08:04:26.724865 (+ 114us) write_op.cc:270] Start()
0623 08:04:26.724913 (+ 48us) write_op.cc:276] Timestamp: P: 1750665863116195 usec, L: 0
0623 08:04:26.725050 (+ 137us) write_op.cc:270] Start()
0623 08:04:26.725069 (+ 19us) write_op.cc:276] Timestamp: P: 1750665863118608 usec, L: 0
0623 08:04:26.725188 (+ 119us) write_op.cc:270] Start()
0623 08:04:26.725205 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863122809 usec, L: 0
0623 08:04:26.725320 (+ 115us) write_op.cc:270] Start()
0623 08:04:26.725342 (+ 22us) write_op.cc:276] Timestamp: P: 1750665863125874 usec, L: 0
0623 08:04:26.725466 (+ 124us) write_op.cc:270] Start()
0623 08:04:26.725483 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863130413 usec, L: 0
0623 08:04:26.725593 (+ 110us) write_op.cc:270] Start()
0623 08:04:26.725621 (+ 28us) write_op.cc:276] Timestamp: P: 1750665863131118 usec, L: 0
0623 08:04:26.725743 (+ 122us) write_op.cc:270] Start()
0623 08:04:26.725775 (+ 32us) write_op.cc:276] Timestamp: P: 1750665863163839 usec, L: 0
0623 08:04:26.725881 (+ 106us) write_op.cc:270] Start()
0623 08:04:26.725898 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863178197 usec, L: 0
0623 08:04:26.750331 (+ 24433us) write_op.cc:270] Start()
0623 08:04:26.750377 (+ 46us) write_op.cc:276] Timestamp: P: 1750665863185348 usec, L: 0
0623 08:04:26.750615 (+ 238us) write_op.cc:270] Start()
0623 08:04:26.750644 (+ 29us) write_op.cc:276] Timestamp: P: 1750665863209348 usec, L: 0
0623 08:04:26.750825 (+ 181us) write_op.cc:270] Start()
0623 08:04:26.750847 (+ 22us) write_op.cc:276] Timestamp: P: 1750665863215881 usec, L: 0
0623 08:04:26.755333 (+ 4486us) write_op.cc:270] Start()
0623 08:04:26.755360 (+ 27us) write_op.cc:276] Timestamp: P: 1750665863217003 usec, L: 0
0623 08:04:26.755684 (+ 324us) write_op.cc:270] Start()
0623 08:04:26.755711 (+ 27us) write_op.cc:276] Timestamp: P: 1750665863288600 usec, L: 0
0623 08:04:26.755959 (+ 248us) write_op.cc:270] Start()
0623 08:04:26.755985 (+ 26us) write_op.cc:276] Timestamp: P: 1750665863294489 usec, L: 0
0623 08:04:26.756151 (+ 166us) write_op.cc:270] Start()
0623 08:04:26.756172 (+ 21us) write_op.cc:276] Timestamp: P: 1750665863296140 usec, L: 0
0623 08:04:26.756298 (+ 126us) write_op.cc:270] Start()
0623 08:04:26.756319 (+ 21us) write_op.cc:276] Timestamp: P: 1750665863301995 usec, L: 0
0623 08:04:26.756468 (+ 149us) write_op.cc:270] Start()
0623 08:04:26.756487 (+ 19us) write_op.cc:276] Timestamp: P: 1750665863311124 usec, L: 0
0623 08:04:26.756629 (+ 142us) write_op.cc:270] Start()
0623 08:04:26.756654 (+ 25us) write_op.cc:276] Timestamp: P: 1750665863315826 usec, L: 0
0623 08:04:26.756808 (+ 154us) write_op.cc:270] Start()
0623 08:04:26.756828 (+ 20us) write_op.cc:276] Timestamp: P: 1750665863335184 usec, L: 0
0623 08:04:26.756967 (+ 139us) write_op.cc:270] Start()
0623 08:04:26.756986 (+ 19us) write_op.cc:276] Timestamp: P: 1750665863336301 usec, L: 0
0623 08:04:26.757141 (+ 155us) write_op.cc:270] Start()
0623 08:04:26.757161 (+ 20us) write_op.cc:276] Timestamp: P: 1750665863338088 usec, L: 0
0623 08:04:26.757270 (+ 109us) write_op.cc:270] Start()
0623 08:04:26.757286 (+ 16us) write_op.cc:276] Timestamp: P: 1750665863364852 usec, L: 0
0623 08:04:26.757414 (+ 128us) write_op.cc:270] Start()
0623 08:04:26.757435 (+ 21us) write_op.cc:276] Timestamp: P: 1750665863368399 usec, L: 0
0623 08:04:26.757573 (+ 138us) write_op.cc:270] Start()
0623 08:04:26.757627 (+ 54us) write_op.cc:276] Timestamp: P: 1750665863369168 usec, L: 0
0623 08:04:26.757765 (+ 138us) write_op.cc:270] Start()
0623 08:04:26.757817 (+ 52us) write_op.cc:276] Timestamp: P: 1750665863371841 usec, L: 0
0623 08:04:26.757962 (+ 145us) write_op.cc:270] Start()
0623 08:04:26.757980 (+ 18us) write_op.cc:276] Timestamp: P: 1750665863374109 usec, L: 0
0623 08:04:26.758125 (+ 145us) write_op.cc:270] Start()
0623 08:04:26.758146 (+ 21us) write_op.cc:276] Timestamp: P: 1750665863387021 usec, L: 0
0623 08:04:26.758287 (+ 141us) write_op.cc:270] Start()
0623 08:04:26.758309 (+ 22us) write_op.cc:276] Timestamp: P: 1750665863411969 usec, L: 0
0623 08:04:26.758472 (+ 163us) write_op.cc:270] Start()
0623 08:04:26.758493 (+ 21us) write_op.cc:276] Timestamp: P: 1750665863437338 usec, L: 0
0623 08:04:26.758664 (+ 171us) write_op.cc:270] Start()
0623 08:04:26.758687 (+ 23us) write_op.cc:276] Timestamp: P: 1750665863438580 usec, L: 0
0623 08:04:26.758864 (+ 177us) write_op.cc:270] Start()
0623 08:04:26.758886 (+ 22us) write_op.cc:276] Timestamp: P: 1750665863455546 usec, L: 0
0623 08:04:26.759023 (+ 137us) write_op.cc:270] Start()
0623 08:04:26.759042 (+ 19us) write_op.cc:276] Timestamp: P: 1750665863456184 usec, L: 0
0623 08:04:26.768274 (+ 9232us) write_op.cc:270] Start()
0623 08:04:26.768416 (+ 142us) write_op.cc:276] Timestamp: P: 1750665863458267 usec, L: 0
0623 08:04:26.769312 (+ 896us) write_op.cc:270] Start()
0623 08:04:26.769337 (+ 25us) write_op.cc:276] Timestamp: P: 1750665863459342 usec, L: 0
0623 08:04:26.769462 (+ 125us) write_op.cc:270] Start()
0623 08:04:26.769479 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863468040 usec, L: 0
0623 08:04:26.769596 (+ 117us) write_op.cc:270] Start()
0623 08:04:26.769620 (+ 24us) write_op.cc:276] Timestamp: P: 1750665863469268 usec, L: 0
0623 08:04:26.769747 (+ 127us) write_op.cc:270] Start()
0623 08:04:26.769762 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863470126 usec, L: 0
0623 08:04:26.769882 (+ 120us) write_op.cc:270] Start()
0623 08:04:26.769897 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863511806 usec, L: 0
0623 08:04:26.770047 (+ 150us) write_op.cc:270] Start()
0623 08:04:26.770066 (+ 19us) write_op.cc:276] Timestamp: P: 1750665863527792 usec, L: 0
0623 08:04:26.770204 (+ 138us) write_op.cc:270] Start()
0623 08:04:26.770220 (+ 16us) write_op.cc:276] Timestamp: P: 1750665863530409 usec, L: 0
0623 08:04:26.770334 (+ 114us) write_op.cc:270] Start()
0623 08:04:26.770349 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863530973 usec, L: 0
0623 08:04:26.770472 (+ 123us) write_op.cc:270] Start()
0623 08:04:26.770486 (+ 14us) write_op.cc:276] Timestamp: P: 1750665863549128 usec, L: 0
0623 08:04:26.770601 (+ 115us) write_op.cc:270] Start()
0623 08:04:26.770646 (+ 45us) write_op.cc:276] Timestamp: P: 1750665863576733 usec, L: 0
0623 08:04:26.770836 (+ 190us) write_op.cc:270] Start()
0623 08:04:26.770858 (+ 22us) write_op.cc:276] Timestamp: P: 1750665863618220 usec, L: 0
0623 08:04:26.770977 (+ 119us) write_op.cc:270] Start()
0623 08:04:26.770994 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863618871 usec, L: 0
0623 08:04:26.771138 (+ 144us) write_op.cc:270] Start()
0623 08:04:26.771158 (+ 20us) write_op.cc:276] Timestamp: P: 1750665863633543 usec, L: 0
0623 08:04:26.771273 (+ 115us) write_op.cc:270] Start()
0623 08:04:26.771288 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863634221 usec, L: 0
0623 08:04:26.771392 (+ 104us) write_op.cc:270] Start()
0623 08:04:26.771407 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863634878 usec, L: 0
0623 08:04:26.771498 (+ 91us) write_op.cc:270] Start()
0623 08:04:26.771513 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863635494 usec, L: 0
0623 08:04:26.771630 (+ 117us) write_op.cc:270] Start()
0623 08:04:26.771646 (+ 16us) write_op.cc:276] Timestamp: P: 1750665863636053 usec, L: 0
0623 08:04:26.771749 (+ 103us) write_op.cc:270] Start()
0623 08:04:26.771763 (+ 14us) write_op.cc:276] Timestamp: P: 1750665863636666 usec, L: 0
0623 08:04:26.771866 (+ 103us) write_op.cc:270] Start()
0623 08:04:26.771882 (+ 16us) write_op.cc:276] Timestamp: P: 1750665863643258 usec, L: 0
0623 08:04:26.772008 (+ 126us) write_op.cc:270] Start()
0623 08:04:26.772024 (+ 16us) write_op.cc:276] Timestamp: P: 1750665863664775 usec, L: 0
0623 08:04:26.772125 (+ 101us) write_op.cc:270] Start()
0623 08:04:26.772139 (+ 14us) write_op.cc:276] Timestamp: P: 1750665863700975 usec, L: 0
0623 08:04:26.772242 (+ 103us) write_op.cc:270] Start()
0623 08:04:26.772256 (+ 14us) write_op.cc:276] Timestamp: P: 1750665863709644 usec, L: 0
0623 08:04:26.772368 (+ 112us) write_op.cc:270] Start()
0623 08:04:26.772385 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863722961 usec, L: 0
0623 08:04:26.772490 (+ 105us) write_op.cc:270] Start()
0623 08:04:26.772504 (+ 14us) write_op.cc:276] Timestamp: P: 1750665863723806 usec, L: 0
0623 08:04:26.772619 (+ 115us) write_op.cc:270] Start()
0623 08:04:26.772636 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863724514 usec, L: 0
0623 08:04:26.772750 (+ 114us) write_op.cc:270] Start()
0623 08:04:26.772766 (+ 16us) write_op.cc:276] Timestamp: P: 1750665863736692 usec, L: 0
0623 08:04:26.772898 (+ 132us) write_op.cc:270] Start()
0623 08:04:26.772918 (+ 20us) write_op.cc:276] Timestamp: P: 1750665863749318 usec, L: 0
0623 08:04:26.773058 (+ 140us) write_op.cc:270] Start()
0623 08:04:26.773075 (+ 17us) write_op.cc:276] Timestamp: P: 1750665863753835 usec, L: 0
0623 08:04:26.773186 (+ 111us) write_op.cc:270] Start()
0623 08:04:26.773201 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863760397 usec, L: 0
0623 08:04:26.773300 (+ 99us) write_op.cc:270] Start()
0623 08:04:26.773315 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863772423 usec, L: 0
0623 08:04:26.773446 (+ 131us) write_op.cc:270] Start()
0623 08:04:26.773461 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863860661 usec, L: 0
0623 08:04:26.773564 (+ 103us) write_op.cc:270] Start()
0623 08:04:26.773579 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863883829 usec, L: 0
0623 08:04:26.773692 (+ 113us) write_op.cc:270] Start()
0623 08:04:26.773707 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863884702 usec, L: 0
0623 08:04:26.773805 (+ 98us) write_op.cc:270] Start()
0623 08:04:26.773820 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863914887 usec, L: 0
0623 08:04:26.773941 (+ 121us) write_op.cc:270] Start()
0623 08:04:26.773957 (+ 16us) write_op.cc:276] Timestamp: P: 1750665863918429 usec, L: 0
0623 08:04:26.774089 (+ 132us) write_op.cc:270] Start()
0623 08:04:26.774105 (+ 16us) write_op.cc:276] Timestamp: P: 1750665863932680 usec, L: 0
0623 08:04:26.774198 (+ 93us) write_op.cc:270] Start()
0623 08:04:26.774213 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863933552 usec, L: 0
0623 08:04:26.774315 (+ 102us) write_op.cc:270] Start()
0623 08:04:26.774329 (+ 14us) write_op.cc:276] Timestamp: P: 1750665863935682 usec, L: 0
0623 08:04:26.774439 (+ 110us) write_op.cc:270] Start()
0623 08:04:26.774461 (+ 22us) write_op.cc:276] Timestamp: P: 1750665863941101 usec, L: 0
0623 08:04:26.774569 (+ 108us) write_op.cc:270] Start()
0623 08:04:26.774585 (+ 16us) write_op.cc:276] Timestamp: P: 1750665863942366 usec, L: 0
0623 08:04:26.774704 (+ 119us) write_op.cc:270] Start()
0623 08:04:26.774719 (+ 15us) write_op.cc:276] Timestamp: P: 1750665863944775 usec, L: 0
0623 08:04:26.779531 (+ 4812us) write_op.cc:270] Start()
0623 08:04:26.779562 (+ 31us) write_op.cc:276] Timestamp: P: 1750665863997087 usec, L: 0
0623 08:04:26.779767 (+ 205us) write_op.cc:270] Start()
0623 08:04:26.779789 (+ 22us) write_op.cc:276] Timestamp: P: 1750665863998079 usec, L: 0
0623 08:04:26.779954 (+ 165us) write_op.cc:270] Start()
0623 08:04:26.779980 (+ 26us) write_op.cc:276] Timestamp: P: 1750665864071253 usec, L: 0
0623 08:04:26.780125 (+ 145us) write_op.cc:270] Start()
0623 08:04:26.780143 (+ 18us) write_op.cc:276] Timestamp: P: 1750665864073275 usec, L: 0
0623 08:04:26.780293 (+ 150us) write_op.cc:270] Start()
0623 08:04:26.780314 (+ 21us) write_op.cc:276] Timestamp: P: 1750665864075146 usec, L: 0
0623 08:04:26.795384 (+ 15070us) write_op.cc:270] Start()
0623 08:04:26.795429 (+ 45us) write_op.cc:276] Timestamp: P: 1750665864076015 usec, L: 0
0623 08:04:26.796213 (+ 784us) write_op.cc:270] Start()
0623 08:04:26.834018 (+ 37805us) write_op.cc:276] Timestamp: P: 1750665864076946 usec, L: 0
0623 08:04:26.834821 (+ 803us) write_op.cc:270] Start()
0623 08:04:26.834853 (+ 32us) write_op.cc:276] Timestamp: P: 1750665864077849 usec, L: 0
0623 08:04:26.835021 (+ 168us) write_op.cc:270] Start()
0623 08:04:26.835038 (+ 17us) write_op.cc:276] Timestamp: P: 1750665864080964 usec, L: 0
0623 08:04:26.835135 (+ 97us) write_op.cc:270] Start()
0623 08:04:26.835149 (+ 14us) write_op.cc:276] Timestamp: P: 1750665864083391 usec, L: 0
0623 08:04:26.837712 (+ 2563us) write_op.cc:270] Start()
0623 08:04:26.837742 (+ 30us) write_op.cc:276] Timestamp: P: 1750665864142093 usec, L: 0
0623 08:04:26.837888 (+ 146us) write_op.cc:270] Start()
0623 08:04:26.837905 (+ 17us) write_op.cc:276] Timestamp: P: 1750665864144304 usec, L: 0
0623 08:04:26.838055 (+ 150us) write_op.cc:270] Start()
0623 08:04:26.838076 (+ 21us) write_op.cc:276] Timestamp: P: 1750665864209017 usec, L: 0
0623 08:04:26.838211 (+ 135us) write_op.cc:270] Start()
0623 08:04:26.838232 (+ 21us) write_op.cc:276] Timestamp: P: 1750665864209763 usec, L: 0
0623 08:04:26.838348 (+ 116us) write_op.cc:270] Start()
0623 08:04:26.838370 (+ 22us) write_op.cc:276] Timestamp: P: 1750665864210467 usec, L: 0
0623 08:04:26.838542 (+ 172us) write_op.cc:270] Start()
0623 08:04:26.838562 (+ 20us) write_op.cc:276] Timestamp: P: 1750665864272659 usec, L: 0
0623 08:04:26.838708 (+ 146us) write_op.cc:270] Start()
0623 08:04:26.838725 (+ 17us) write_op.cc:276] Timestamp: P: 1750665864274085 usec, L: 0
0623 08:04:26.842776 (+ 4051us) write_op.cc:270] Start()
0623 08:04:26.842802 (+ 26us) write_op.cc:276] Timestamp: P: 1750665864275336 usec, L: 0
0623 08:04:26.843028 (+ 226us) write_op.cc:270] Start()
0623 08:04:26.843045 (+ 17us) write_op.cc:276] Timestamp: P: 1750665864276510 usec, L: 0
0623 08:04:26.843178 (+ 133us) write_op.cc:270] Start()
0623 08:04:26.843198 (+ 20us) write_op.cc:276] Timestamp: P: 1750665864277746 usec, L: 0
0623 08:04:26.843318 (+ 120us) write_op.cc:270] Start()
0623 08:04:26.843356 (+ 38us) write_op.cc:276] Timestamp: P: 1750665864442375 usec, L: 0
0623 08:04:26.843468 (+ 112us) write_op.cc:270] Start()
0623 08:04:26.843484 (+ 16us) write_op.cc:276] Timestamp: P: 1750665864443764 usec, L: 0
0623 08:04:26.843589 (+ 105us) write_op.cc:270] Start()
0623 08:04:26.843617 (+ 28us) write_op.cc:276] Timestamp: P: 1750665864444989 usec, L: 0
0623 08:04:26.861639 (+ 18022us) write_op.cc:270] Start()
0623 08:04:26.861694 (+ 55us) write_op.cc:276] Timestamp: P: 1750665864446252 usec, L: 0
0623 08:04:26.861896 (+ 202us) write_op.cc:270] Start()
0623 08:04:26.861918 (+ 22us) write_op.cc:276] Timestamp: P: 1750665864447438 usec, L: 0
0623 08:04:26.862044 (+ 126us) write_op.cc:270] Start()
0623 08:04:26.862064 (+ 20us) write_op.cc:276] Timestamp: P: 1750665864552581 usec, L: 0
0623 08:04:26.862204 (+ 140us) write_op.cc:270] Start()
0623 08:04:26.862224 (+ 20us) write_op.cc:276] Timestamp: P: 1750665864611945 usec, L: 0
0623 08:04:26.913621 (+ 51397us) write_op.cc:270] Start()
0623 08:04:26.913664 (+ 43us) write_op.cc:276] Timestamp: P: 1750665864628163 usec, L: 0
0623 08:04:26.913905 (+ 241us) write_op.cc:270] Start()
0623 08:04:26.913929 (+ 24us) write_op.cc:276] Timestamp: P: 1750665864634647 usec, L: 0
0623 08:04:26.914074 (+ 145us) write_op.cc:270] Start()
0623 08:04:26.914092 (+ 18us) write_op.cc:276] Timestamp: P: 1750665864666856 usec, L: 0
0623 08:04:26.914210 (+ 118us) write_op.cc:270] Start()
0623 08:04:26.914234 (+ 24us) write_op.cc:276] Timestamp: P: 1750665864682372 usec, L: 0
0623 08:04:26.915759 (+ 1525us) write_op.cc:270] Start()
0623 08:04:26.915786 (+ 27us) write_op.cc:276] Timestamp: P: 1750665864684765 usec, L: 0
0623 08:04:26.915949 (+ 163us) write_op.cc:270] Start()
0623 08:04:26.915974 (+ 25us) write_op.cc:276] Timestamp: P: 1750665864685907 usec, L: 0
0623 08:04:26.927091 (+ 11117us) write_op.cc:270] Start()
0623 08:04:26.927129 (+ 38us) write_op.cc:276] Timestamp: P: 1750665864687578 usec, L: 0
0623 08:04:26.927320 (+ 191us) write_op.cc:270] Start()
0623 08:04:26.927339 (+ 19us) write_op.cc:276] Timestamp: P: 1750665864780802 usec, L: 0
0623 08:04:26.927492 (+ 153us) write_op.cc:270] Start()
0623 08:04:26.927511 (+ 19us) write_op.cc:276] Timestamp: P: 1750665864793726 usec, L: 0
0623 08:04:26.927659 (+ 148us) write_op.cc:270] Start()
0623 08:04:26.927679 (+ 20us) write_op.cc:276] Timestamp: P: 1750665864798090 usec, L: 0
0623 08:04:26.927811 (+ 132us) write_op.cc:270] Start()
0623 08:04:26.927831 (+ 20us) write_op.cc:276] Timestamp: P: 1750665864802505 usec, L: 0
0623 08:04:26.927945 (+ 114us) write_op.cc:270] Start()
0623 08:04:26.927962 (+ 17us) write_op.cc:276] Timestamp: P: 1750665864807240 usec, L: 0
0623 08:04:26.928069 (+ 107us) write_op.cc:270] Start()
0623 08:04:26.928086 (+ 17us) write_op.cc:276] Timestamp: P: 1750665864860169 usec, L: 0
0623 08:04:26.928223 (+ 137us) write_op.cc:270] Start()
0623 08:04:26.928240 (+ 17us) write_op.cc:276] Timestamp: P: 1750665864861347 usec, L: 0
0623 08:04:26.928352 (+ 112us) write_op.cc:270] Start()
0623 08:04:26.928377 (+ 25us) write_op.cc:276] Timestamp: P: 1750665864862025 usec, L: 0
0623 08:04:26.928485 (+ 108us) write_op.cc:270] Start()
0623 08:04:26.928505 (+ 20us) write_op.cc:276] Timestamp: P: 1750665864862829 usec, L: 0
0623 08:04:26.928691 (+ 186us) write_op.cc:270] Start()
0623 08:04:26.928712 (+ 21us) write_op.cc:276] Timestamp: P: 1750665864863591 usec, L: 0
0623 08:04:26.928851 (+ 139us) write_op.cc:270] Start()
0623 08:04:26.928869 (+ 18us) write_op.cc:276] Timestamp: P: 1750665864864626 usec, L: 0
0623 08:04:26.929019 (+ 150us) write_op.cc:270] Start()
0623 08:04:26.929041 (+ 22us) write_op.cc:276] Timestamp: P: 1750665864865899 usec, L: 0
0623 08:04:26.929211 (+ 170us) write_op.cc:270] Start()
0623 08:04:26.929235 (+ 24us) write_op.cc:276] Timestamp: P: 1750665864867091 usec, L: 0
0623 08:04:26.929387 (+ 152us) write_op.cc:270] Start()
0623 08:04:26.929408 (+ 21us) write_op.cc:276] Timestamp: P: 1750665864887864 usec, L: 0
0623 08:04:26.929547 (+ 139us) write_op.cc:270] Start()
0623 08:04:26.929568 (+ 21us) write_op.cc:276] Timestamp: P: 1750665864907688 usec, L: 0
0623 08:04:26.929716 (+ 148us) write_op.cc:270] Start()
0623 08:04:26.929735 (+ 19us) write_op.cc:276] Timestamp: P: 1750665864909227 usec, L: 0
0623 08:04:26.929880 (+ 145us) write_op.cc:270] Start()
0623 08:04:26.929899 (+ 19us) write_op.cc:276] Timestamp: P: 1750665864917587 usec, L: 0
0623 08:04:26.930035 (+ 136us) write_op.cc:270] Start()
0623 08:04:26.930054 (+ 19us) write_op.cc:276] Timestamp: P: 1750665864920995 usec, L: 0
0623 08:04:26.930208 (+ 154us) write_op.cc:270] Start()
0623 08:04:26.930230 (+ 22us) write_op.cc:276] Timestamp: P: 1750665864931968 usec, L: 0
0623 08:04:26.930366 (+ 136us) write_op.cc:270] Start()
0623 08:04:26.930385 (+ 19us) write_op.cc:276] Timestamp: P: 1750665864938888 usec, L: 0
0623 08:04:26.930506 (+ 121us) write_op.cc:270] Start()
0623 08:04:26.930530 (+ 24us) write_op.cc:276] Timestamp: P: 1750665864940386 usec, L: 0
0623 08:04:26.930677 (+ 147us) write_op.cc:270] Start()
0623 08:04:26.930695 (+ 18us) write_op.cc:276] Timestamp: P: 1750665864956490 usec, L: 0
0623 08:04:26.939733 (+ 9038us) write_op.cc:270] Start()
0623 08:04:26.939830 (+ 97us) write_op.cc:276] Timestamp: P: 1750665865022853 usec, L: 0
0623 08:04:26.940024 (+ 194us) write_op.cc:270] Start()
0623 08:04:26.940044 (+ 20us) write_op.cc:276] Timestamp: P: 1750665865037203 usec, L: 0
0623 08:04:26.940175 (+ 131us) write_op.cc:270] Start()
0623 08:04:26.940193 (+ 18us) write_op.cc:276] Timestamp: P: 1750665865041287 usec, L: 0
0623 08:04:26.940305 (+ 112us) write_op.cc:270] Start()
0623 08:04:26.940321 (+ 16us) write_op.cc:276] Timestamp: P: 1750665865049642 usec, L: 0
0623 08:04:26.940461 (+ 140us) write_op.cc:270] Start()
0623 08:04:26.940479 (+ 18us) write_op.cc:276] Timestamp: P: 1750665865082364 usec, L: 0
0623 08:04:26.940605 (+ 126us) write_op.cc:270] Start()
0623 08:04:26.940624 (+ 19us) write_op.cc:276] Timestamp: P: 1750665865084682 usec, L: 0
0623 08:04:26.940746 (+ 122us) write_op.cc:270] Start()
0623 08:04:26.940763 (+ 17us) write_op.cc:276] Timestamp: P: 1750665865087784 usec, L: 0
0623 08:04:26.940904 (+ 141us) write_op.cc:270] Start()
0623 08:04:26.940924 (+ 20us) write_op.cc:276] Timestamp: P: 1750665865089477 usec, L: 0
0623 08:04:26.941069 (+ 145us) write_op.cc:270] Start()
0623 08:04:26.941087 (+ 18us) write_op.cc:276] Timestamp: P: 1750665865090026 usec, L: 0
0623 08:04:26.941206 (+ 119us) write_op.cc:270] Start()
0623 08:04:26.941222 (+ 16us) write_op.cc:276] Timestamp: P: 1750665865091153 usec, L: 0
0623 08:04:26.941367 (+ 145us) write_op.cc:270] Start()
0623 08:04:26.941384 (+ 17us) write_op.cc:276] Timestamp: P: 1750665865092130 usec, L: 0
0623 08:04:26.941494 (+ 110us) write_op.cc:270] Start()
0623 08:04:26.941510 (+ 16us) write_op.cc:276] Timestamp: P: 1750665865138049 usec, L: 0
0623 08:04:26.941633 (+ 123us) write_op.cc:270] Start()
0623 08:04:26.941650 (+ 17us) write_op.cc:276] Timestamp: P: 1750665865139979 usec, L: 0
0623 08:04:26.941769 (+ 119us) write_op.cc:270] Start()
0623 08:04:26.941788 (+ 19us) write_op.cc:276] Timestamp: P: 1750665865149288 usec, L: 0
0623 08:04:26.941922 (+ 134us) write_op.cc:270] Start()
0623 08:04:26.941940 (+ 18us) write_op.cc:276] Timestamp: P: 1750665865149912 usec, L: 0
0623 08:04:26.942073 (+ 133us) write_op.cc:270] Start()
0623 08:04:26.942095 (+ 22us) write_op.cc:276] Timestamp: P: 1750665865150455 usec, L: 0
0623 08:04:26.942246 (+ 151us) write_op.cc:270] Start()
0623 08:04:26.942266 (+ 20us) write_op.cc:276] Timestamp: P: 1750665865151037 usec, L: 0
0623 08:04:26.955098 (+ 12832us) write_op.cc:270] Start()
0623 08:04:26.955132 (+ 34us) write_op.cc:276] Timestamp: P: 1750665865161390 usec, L: 0
0623 08:04:26.955299 (+ 167us) write_op.cc:270] Start()
0623 08:04:26.955319 (+ 20us) write_op.cc:276] Timestamp: P: 1750665865239082 usec, L: 0
0623 08:04:26.955485 (+ 166us) write_op.cc:270] Start()
0623 08:04:26.955506 (+ 21us) write_op.cc:276] Timestamp: P: 1750665865240286 usec, L: 0
0623 08:04:26.955660 (+ 154us) write_op.cc:270] Start()
0623 08:04:26.955681 (+ 21us) write_op.cc:276] Timestamp: P: 1750665865242902 usec, L: 0
0623 08:04:26.955817 (+ 136us) write_op.cc:270] Start()
0623 08:04:26.955836 (+ 19us) write_op.cc:276] Timestamp: P: 1750665865268047 usec, L: 0
0623 08:04:26.955977 (+ 141us) write_op.cc:270] Start()
0623 08:04:26.955998 (+ 21us) write_op.cc:276] Timestamp: P: 1750665865272322 usec, L: 0
0623 08:04:26.956138 (+ 140us) write_op.cc:270] Start()
0623 08:04:26.956159 (+ 21us) write_op.cc:276] Timestamp: P: 1750665865273015 usec, L: 0
0623 08:04:26.956297 (+ 138us) write_op.cc:270] Start()
0623 08:04:26.956317 (+ 20us) write_op.cc:276] Timestamp: P: 1750665865276334 usec, L: 0
0623 08:04:26.956490 (+ 173us) write_op.cc:270] Start()
0623 08:04:26.956512 (+ 22us) write_op.cc:276] Timestamp: P: 1750665865286094 usec, L: 0
0623 08:04:26.956667 (+ 155us) write_op.cc:270] Start()
0623 08:04:26.956687 (+ 20us) write_op.cc:276] Timestamp: P: 1750665865303429 usec, L: 0
0623 08:04:26.956815 (+ 128us) write_op.cc:270] Start()
0623 08:04:26.956834 (+ 19us) write_op.cc:276] Timestamp: P: 1750665865344146 usec, L: 0
0623 08:04:26.956979 (+ 145us) write_op.cc:270] Start()
0623 08:04:26.956999 (+ 20us) write_op.cc:276] Timestamp: P: 1750665865346627 usec, L: 0
0623 08:04:26.957125 (+ 126us) write_op.cc:270] Start()
0623 08:04:26.957145 (+ 20us) write_op.cc:276] Timestamp: P: 1750665865356199 usec, L: 0
0623 08:04:26.957281 (+ 136us) write_op.cc:270] Start()
0623 08:04:26.957303 (+ 22us) write_op.cc:276] Timestamp: P: 1750665
I20250623 08:04:28.267421 339 tablet_bootstrap.cc:492] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.267571 339 ts_tablet_manager.cc:1397] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.014s
I20250623 08:04:28.268257 339 raft_consensus.cc:357] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.268383 339 raft_consensus.cc:383] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.268445 339 raft_consensus.cc:738] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.268599 339 consensus_queue.cc:260] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.268831 685 raft_consensus.cc:491] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.268910 32379 raft_consensus.cc:2466] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:28.268999 685 raft_consensus.cc:513] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.269098 339 ts_tablet_manager.cc:1428] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:28.269387 339 tablet_bootstrap.cc:492] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.269695 685 leader_election.cc:290] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.271059 339 tablet_bootstrap.cc:654] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.272111 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9918b15e48144ef0b8f3aedc2135be59" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:28.272702 32314 leader_election.cc:343] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:28.272747 32453 leader_election.cc:343] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.272914 339 tablet_bootstrap.cc:492] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.273039 339 ts_tablet_manager.cc:1397] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250623 08:04:28.273092 32453 leader_election.cc:304] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.273687 344 ts_tablet_manager.cc:1918] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:28.273676 339 raft_consensus.cc:357] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.273841 339 raft_consensus.cc:383] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.273854 344 log.cc:1199] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/228375592aee48a1aa21e1d16962b663
I20250623 08:04:28.273908 339 raft_consensus.cc:738] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.274134 339 consensus_queue.cc:260] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.274689 339 ts_tablet_manager.cc:1428] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:28.274940 344 ts_tablet_manager.cc:1939] T 228375592aee48a1aa21e1d16962b663 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:28.275058 339 tablet_bootstrap.cc:492] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.276274 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9918b15e48144ef0b8f3aedc2135be59" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.276491 32517 raft_consensus.cc:2466] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.276914 32314 leader_election.cc:304] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.277143 342 tablet_bootstrap.cc:654] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.277325 685 raft_consensus.cc:2802] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.277418 685 raft_consensus.cc:491] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.277529 685 raft_consensus.cc:3058] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.278573 699 raft_consensus.cc:513] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.279217 697 raft_consensus.cc:2802] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.279294 32520 raft_consensus.cc:2391] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:28.279383 697 raft_consensus.cc:695] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:28.279837 344 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55518 (request call id 230) took 1106 ms. Trace:
I20250623 08:04:28.285094 32653 raft_consensus.cc:3058] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.288807 700 raft_consensus.cc:695] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [term 3 LEADER]: Becoming Leader. State: Replica: d31835ec4ad74332a00998d431492466, State: Running, Role: LEADER
I20250623 08:04:28.289253 714 raft_consensus.cc:491] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.289311 700 consensus_queue.cc:237] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 2.356, Last appended by leader: 356, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:28.289669 714 raft_consensus.cc:513] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.289649 641 consensus_queue.cc:1035] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:28.291929 32314 leader_election.cc:304] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:28.292409 714 leader_election.cc:290] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:28.292454 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db9480bb07343fe935fe9bfc98bca77" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.292963 727 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d31835ec4ad74332a00998d431492466. Latest consensus state: current_term: 3 leader_uuid: "d31835ec4ad74332a00998d431492466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:28.293555 341 ts_tablet_manager.cc:1918] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250623 08:04:28.295174 727 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: This master's current role is: LEADER
I20250623 08:04:28.297220 641 raft_consensus.cc:2747] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.300318 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.300612 341 log.cc:1199] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/7300f526d91f441bb164fe47f3a4f0ce
I20250623 08:04:28.300695 32381 raft_consensus.cc:2391] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:28.301548 685 raft_consensus.cc:513] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.302186 685 leader_election.cc:290] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.303303 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9918b15e48144ef0b8f3aedc2135be59" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.303542 339 tablet_bootstrap.cc:654] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:28.303824 32453 consensus_peers.cc:487] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce1f3ef3690c4837b95b105733884f1f. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:28.304531 32314 leader_election.cc:343] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.304769 697 consensus_queue.cc:237] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.308023 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
W20250623 08:04:28.308651 32453 leader_election.cc:343] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.309387 32453 leader_election.cc:304] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.311928 657 raft_consensus.cc:2747] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.312502 343 tablet_bootstrap.cc:492] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.312705 343 ts_tablet_manager.cc:1397] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.108s user 0.002s sys 0.002s
I20250623 08:04:28.313803 699 leader_election.cc:290] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.313762 343 raft_consensus.cc:357] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.313988 343 raft_consensus.cc:383] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.314116 343 raft_consensus.cc:738] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.305896 344 rpcz_store.cc:276] 0623 08:04:27.169136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:27.169180 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:28.084953 (+915773us) ts_tablet_manager.cc:1145] Deleting tablet 228375592aee48a1aa21e1d16962b663
0623 08:04:28.084962 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:28.130170 (+ 45208us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:28.257660 (+127490us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:28.276047 (+ 18387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":915616}
I20250623 08:04:28.314666 341 ts_tablet_manager.cc:1939] T 7300f526d91f441bb164fe47f3a4f0ce P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:28.315595 344 tablet_replica.cc:331] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:28.315687 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9918b15e48144ef0b8f3aedc2135be59" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.315858 344 raft_consensus.cc:2241] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:28.315999 662 raft_consensus.cc:513] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.319840 343 consensus_queue.cc:260] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.320544 343 ts_tablet_manager.cc:1428] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250623 08:04:28.322135 343 tablet_bootstrap.cc:492] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.322975 685 raft_consensus.cc:491] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.323153 685 raft_consensus.cc:513] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.323781 685 leader_election.cc:290] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.324488 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726d80e05a344f27b5a289723cb5aed7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:28.325028 32314 leader_election.cc:343] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.326514 343 tablet_bootstrap.cc:654] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.327963 344 raft_consensus.cc:2270] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:28.329092 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726d80e05a344f27b5a289723cb5aed7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.329301 339 tablet_bootstrap.cc:492] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.329344 32517 raft_consensus.cc:2466] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.329439 339 ts_tablet_manager.cc:1397] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.019s
I20250623 08:04:28.329876 32314 leader_election.cc:304] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.330233 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0db9480bb07343fe935fe9bfc98bca77" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.330299 685 raft_consensus.cc:2802] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.330410 685 raft_consensus.cc:491] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.330520 685 raft_consensus.cc:3058] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.330734 344 ts_tablet_manager.cc:1905] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:28.331135 32518 raft_consensus.cc:3058] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.332885 685 raft_consensus.cc:513] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.333504 685 leader_election.cc:290] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.333853 32589 leader_election.cc:304] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.330334 339 raft_consensus.cc:357] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.334139 339 raft_consensus.cc:383] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.334272 339 raft_consensus.cc:738] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.334546 339 consensus_queue.cc:260] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.334565 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726d80e05a344f27b5a289723cb5aed7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.334885 710 raft_consensus.cc:2747] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 08:04:28.335264 32314 leader_election.cc:343] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.335611 339 ts_tablet_manager.cc:1428] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250623 08:04:28.336010 339 tablet_bootstrap.cc:492] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.336119 32518 raft_consensus.cc:2466] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.338094 339 tablet_bootstrap.cc:654] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.351995 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "776b412b3f564a7581c6a5e1c84e8499" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.352119 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "776b412b3f564a7581c6a5e1c84e8499" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.352324 32381 raft_consensus.cc:3058] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.353549 343 tablet_bootstrap.cc:492] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.353718 343 ts_tablet_manager.cc:1397] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250623 08:04:28.354493 343 raft_consensus.cc:357] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.354691 343 raft_consensus.cc:383] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.354784 343 raft_consensus.cc:738] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.355008 343 consensus_queue.cc:260] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.355268 32381 raft_consensus.cc:2466] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:28.355571 343 ts_tablet_manager.cc:1428] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:28.355911 343 tablet_bootstrap.cc:492] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.357684 343 tablet_bootstrap.cc:654] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.361737 343 tablet_bootstrap.cc:492] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.361953 343 ts_tablet_manager.cc:1397] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20250623 08:04:28.362792 343 raft_consensus.cc:357] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.363075 343 raft_consensus.cc:383] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.363219 343 raft_consensus.cc:738] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.363555 343 consensus_queue.cc:260] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.364675 343 ts_tablet_manager.cc:1428] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:28.365056 343 tablet_bootstrap.cc:492] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.367151 343 tablet_bootstrap.cc:654] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.345448 699 consensus_queue.cc:1035] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:28.345837 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726d80e05a344f27b5a289723cb5aed7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.375905 32314 leader_election.cc:304] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.376133 32520 raft_consensus.cc:3058] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.376259 685 raft_consensus.cc:2802] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.376428 685 raft_consensus.cc:695] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:28.376693 685 consensus_queue.cc:237] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.380080 32520 raft_consensus.cc:2466] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.380729 32314 leader_election.cc:304] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.380990 641 raft_consensus.cc:2802] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.381136 641 raft_consensus.cc:695] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:28.381393 641 consensus_queue.cc:237] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.381939 342 tablet_bootstrap.cc:492] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.375317 343 tablet_bootstrap.cc:492] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.382226 343 ts_tablet_manager.cc:1397] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.010s
I20250623 08:04:28.382498 342 ts_tablet_manager.cc:1397] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.126s user 0.005s sys 0.000s
I20250623 08:04:28.383365 343 raft_consensus.cc:357] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:28.357062 32453 leader_election.cc:343] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.384089 32453 leader_election.cc:304] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.384192 344 ts_tablet_manager.cc:1918] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:28.384341 344 log.cc:1199] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/696f5d2797ef463bb72a2119501c6ef3
I20250623 08:04:28.384419 699 raft_consensus.cc:2802] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.384575 699 raft_consensus.cc:695] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:28.384735 344 ts_tablet_manager.cc:1939] T 696f5d2797ef463bb72a2119501c6ef3 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:28.384836 699 consensus_queue.cc:237] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.385504 344 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55518 (request call id 233) took 1157 ms. Trace:
I20250623 08:04:28.385622 344 rpcz_store.cc:276] 0623 08:04:27.227559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:27.227623 (+ 64us) service_pool.cc:225] Handling call
0623 08:04:28.315408 (+1087785us) ts_tablet_manager.cc:1145] Deleting tablet 696f5d2797ef463bb72a2119501c6ef3
0623 08:04:28.315420 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:28.347822 (+ 32402us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:28.384044 (+ 36222us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:28.385446 (+ 1402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1087537}
I20250623 08:04:28.386209 344 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55518 (request call id 234) took 1107 ms. Trace:
I20250623 08:04:28.386322 344 rpcz_store.cc:276] 0623 08:04:27.278421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:27.278481 (+ 60us) service_pool.cc:225] Handling call
0623 08:04:28.386011 (+1107530us) ts_tablet_manager.cc:1145] Deleting tablet 175ee90133994154b7e4b58b0770db3b
0623 08:04:28.386023 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:28.386149 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107288}
I20250623 08:04:28.395262 344 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55518 (request call id 236) took 1115 ms. Trace:
I20250623 08:04:28.395494 344 rpcz_store.cc:276] 0623 08:04:27.279797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:27.279850 (+ 53us) service_pool.cc:225] Handling call
0623 08:04:28.395063 (+1115213us) ts_tablet_manager.cc:1145] Deleting tablet 966a159e212040109e2842c15b69fb71
0623 08:04:28.395074 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:28.395201 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115007}
I20250623 08:04:28.396195 344 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55518 (request call id 235) took 1116 ms. Trace:
I20250623 08:04:28.396368 344 rpcz_store.cc:276] 0623 08:04:27.279681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:27.279981 (+ 300us) service_pool.cc:225] Handling call
0623 08:04:28.396016 (+1116035us) ts_tablet_manager.cc:1145] Deleting tablet 4cb66c7a273a412db3ae77af9e30d036
0623 08:04:28.396028 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:28.396133 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115860}
I20250623 08:04:28.397066 344 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55518 (request call id 237) took 1117 ms. Trace:
I20250623 08:04:28.397236 344 rpcz_store.cc:276] 0623 08:04:27.279932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:27.280081 (+ 149us) service_pool.cc:225] Handling call
0623 08:04:28.396845 (+1116764us) ts_tablet_manager.cc:1145] Deleting tablet e22f9bae5f7248af99cec06ce6dbbd41
0623 08:04:28.396857 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:28.397001 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116580}
I20250623 08:04:28.397848 344 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55518 (request call id 238) took 1060 ms. Trace:
I20250623 08:04:28.398016 344 rpcz_store.cc:276] 0623 08:04:27.337735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:27.337792 (+ 57us) service_pool.cc:225] Handling call
0623 08:04:28.397690 (+1059898us) ts_tablet_manager.cc:1145] Deleting tablet 8dc696236a15420e9666cca708068fb2
0623 08:04:28.397701 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:28.397794 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059692}
I20250623 08:04:28.398625 344 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55518 (request call id 239) took 1028 ms. Trace:
I20250623 08:04:28.398849 344 rpcz_store.cc:276] 0623 08:04:27.370591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:27.370689 (+ 98us) service_pool.cc:225] Handling call
0623 08:04:28.398460 (+1027771us) ts_tablet_manager.cc:1145] Deleting tablet 205da880685b49b3b8669f73dc2c563b
0623 08:04:28.398472 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:28.398561 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1027509}
I20250623 08:04:28.354791 662 leader_election.cc:290] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.399842 344 tablet_replica.cc:331] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:28.400159 344 raft_consensus.cc:2241] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:28.401432 344 raft_consensus.cc:2270] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:28.400164 342 raft_consensus.cc:357] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.401854 342 raft_consensus.cc:383] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.390229 32653 raft_consensus.cc:3058] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.402009 342 raft_consensus.cc:738] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.402326 342 consensus_queue.cc:260] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.402973 342 ts_tablet_manager.cc:1428] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:28.403285 342 tablet_bootstrap.cc:492] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.405009 342 tablet_bootstrap.cc:654] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.405721 339 tablet_bootstrap.cc:492] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.405930 339 ts_tablet_manager.cc:1397] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.070s user 0.014s sys 0.027s
I20250623 08:04:28.408100 662 consensus_queue.cc:1035] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:28.412767 641 consensus_queue.cc:1035] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:28.413601 343 raft_consensus.cc:383] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.413954 343 raft_consensus.cc:738] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.414542 685 raft_consensus.cc:491] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.414541 343 consensus_queue.cc:260] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.414716 685 raft_consensus.cc:513] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.415630 343 ts_tablet_manager.cc:1428] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20250623 08:04:28.415977 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "966a159e212040109e2842c15b69fb71" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.416322 685 leader_election.cc:290] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:28.416576 32314 leader_election.cc:343] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.417905 339 raft_consensus.cc:357] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.418094 339 raft_consensus.cc:383] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.418164 339 raft_consensus.cc:738] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.418417 339 consensus_queue.cc:260] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.418924 339 ts_tablet_manager.cc:1428] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250623 08:04:28.419232 339 tablet_bootstrap.cc:492] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.420783 342 tablet_bootstrap.cc:492] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.421051 342 ts_tablet_manager.cc:1397] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.005s
I20250623 08:04:28.421916 342 raft_consensus.cc:357] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.417086 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "966a159e212040109e2842c15b69fb71" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.432848 339 tablet_bootstrap.cc:654] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.435449 343 tablet_bootstrap.cc:492] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.437095 343 tablet_bootstrap.cc:654] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.436818 662 raft_consensus.cc:491] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.441696 662 raft_consensus.cc:513] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.442853 662 leader_election.cc:290] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.439505 344 ts_tablet_manager.cc:1905] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:28.443202 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e4678ca1c8f498c83c6bc2553882120" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:28.443809 32453 leader_election.cc:343] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.444094 342 raft_consensus.cc:383] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.444567 342 raft_consensus.cc:738] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.445479 343 tablet_bootstrap.cc:492] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.447173 342 consensus_queue.cc:260] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.448087 343 ts_tablet_manager.cc:1397] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20250623 08:04:28.448683 342 ts_tablet_manager.cc:1428] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.027s user 0.000s sys 0.002s
I20250623 08:04:28.448985 342 tablet_bootstrap.cc:492] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.449101 343 raft_consensus.cc:357] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.449366 343 raft_consensus.cc:383] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.449512 343 raft_consensus.cc:738] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.449822 343 consensus_queue.cc:260] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.450374 343 ts_tablet_manager.cc:1428] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:28.451076 343 tablet_bootstrap.cc:492] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.456010 32656 raft_consensus.cc:3058] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:28.456692 32314 leader_election.cc:343] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.458407 32314 leader_election.cc:304] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:28.459014 685 raft_consensus.cc:2747] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.460016 685 consensus_queue.cc:1035] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:28.463389 342 tablet_bootstrap.cc:654] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.465508 343 tablet_bootstrap.cc:654] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.468938 344 ts_tablet_manager.cc:1918] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:28.469146 344 log.cc:1199] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/ce1f3ef3690c4837b95b105733884f1f
I20250623 08:04:28.469532 339 tablet_bootstrap.cc:492] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.469734 339 ts_tablet_manager.cc:1397] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.012s
I20250623 08:04:28.470091 344 ts_tablet_manager.cc:1939] T ce1f3ef3690c4837b95b105733884f1f P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:28.471457 344 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55518 (request call id 240) took 1100 ms. Trace:
I20250623 08:04:28.470664 339 raft_consensus.cc:357] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.472334 339 raft_consensus.cc:383] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.472442 339 raft_consensus.cc:738] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.472985 344 rpcz_store.cc:276] 0623 08:04:27.371224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:27.371286 (+ 62us) service_pool.cc:225] Handling call
0623 08:04:28.399739 (+1028453us) ts_tablet_manager.cc:1145] Deleting tablet ce1f3ef3690c4837b95b105733884f1f
0623 08:04:28.399750 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:28.462263 (+ 62513us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:28.468805 (+ 6542us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:28.471421 (+ 2616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028259}
I20250623 08:04:28.472692 339 consensus_queue.cc:260] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.474821 339 ts_tablet_manager.cc:1428] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:28.475227 339 tablet_bootstrap.cc:492] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.477324 339 tablet_bootstrap.cc:654] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.479068 342 tablet_bootstrap.cc:492] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.479303 342 ts_tablet_manager.cc:1397] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.001s
I20250623 08:04:28.480324 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e4678ca1c8f498c83c6bc2553882120" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:28.480903 32453 leader_election.cc:343] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.481032 32453 leader_election.cc:304] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.481209 344 tablet_replica.cc:331] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:28.481431 344 raft_consensus.cc:2241] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:28.481834 344 raft_consensus.cc:2270] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:28.482827 342 raft_consensus.cc:357] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.483033 342 raft_consensus.cc:383] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.483114 342 raft_consensus.cc:738] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.483330 344 ts_tablet_manager.cc:1905] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:28.483546 342 consensus_queue.cc:260] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.483368 699 raft_consensus.cc:2747] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.483795 641 raft_consensus.cc:491] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.483947 641 raft_consensus.cc:513] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.484565 641 leader_election.cc:290] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.485270 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0117cbb09dd4067836266c9d6a32adf" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.485422 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0117cbb09dd4067836266c9d6a32adf" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:28.486649 32314 leader_election.cc:343] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:28.486872 32314 leader_election.cc:343] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.487001 32314 leader_election.cc:304] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:28.487437 641 raft_consensus.cc:2747] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.487855 342 ts_tablet_manager.cc:1428] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250623 08:04:28.491109 342 tablet_bootstrap.cc:492] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.492905 342 tablet_bootstrap.cc:654] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.494659 339 tablet_bootstrap.cc:492] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.494848 339 ts_tablet_manager.cc:1397] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250623 08:04:28.495605 339 raft_consensus.cc:357] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.495894 339 raft_consensus.cc:383] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.495960 339 raft_consensus.cc:738] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.496181 339 consensus_queue.cc:260] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.496691 339 ts_tablet_manager.cc:1428] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:28.496950 339 tablet_bootstrap.cc:492] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.498934 343 tablet_bootstrap.cc:492] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.499092 339 tablet_bootstrap.cc:654] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.499102 343 ts_tablet_manager.cc:1397] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.004s
I20250623 08:04:28.499858 343 raft_consensus.cc:357] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.500056 343 raft_consensus.cc:383] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.500128 343 raft_consensus.cc:738] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.500403 343 consensus_queue.cc:260] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.501001 343 ts_tablet_manager.cc:1428] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:28.501292 343 tablet_bootstrap.cc:492] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.502131 342 tablet_bootstrap.cc:492] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.502276 342 ts_tablet_manager.cc:1397] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20250623 08:04:28.502975 343 tablet_bootstrap.cc:654] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.503197 342 raft_consensus.cc:357] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.503428 342 raft_consensus.cc:383] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.503536 342 raft_consensus.cc:738] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.503786 342 consensus_queue.cc:260] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:28.503926 32453 consensus_peers.cc:487] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:28.504280 342 ts_tablet_manager.cc:1428] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:28.504596 342 tablet_bootstrap.cc:492] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.508883 342 tablet_bootstrap.cc:654] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.511833 344 ts_tablet_manager.cc:1918] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250623 08:04:28.512048 344 log.cc:1199] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/7300f526d91f441bb164fe47f3a4f0ce
I20250623 08:04:28.512539 344 ts_tablet_manager.cc:1939] T 7300f526d91f441bb164fe47f3a4f0ce P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:28.513729 344 tablet_replica.cc:331] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:28.513969 344 raft_consensus.cc:2241] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:28.514267 344 pending_rounds.cc:70] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:28.514391 344 pending_rounds.cc:77] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727395521683456 op_type: NO_OP noop_request { }
I20250623 08:04:28.514601 344 raft_consensus.cc:2887] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:28.514813 344 raft_consensus.cc:2270] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:28.516245 344 ts_tablet_manager.cc:1905] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:28.517712 343 tablet_bootstrap.cc:492] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.517925 343 ts_tablet_manager.cc:1397] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.005s
I20250623 08:04:28.518791 343 raft_consensus.cc:357] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.519029 343 raft_consensus.cc:383] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.519152 343 raft_consensus.cc:738] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.519404 343 consensus_queue.cc:260] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.519987 343 ts_tablet_manager.cc:1428] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:28.520375 343 tablet_bootstrap.cc:492] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.521219 710 raft_consensus.cc:491] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.521377 710 raft_consensus.cc:513] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.522050 710 leader_election.cc:290] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.522902 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.522936 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.523172 32381 raft_consensus.cc:2391] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:28.523231 32520 raft_consensus.cc:2391] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:28.523808 32589 leader_election.cc:304] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.524235 710 raft_consensus.cc:3058] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.525417 339 tablet_bootstrap.cc:492] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.525597 339 ts_tablet_manager.cc:1397] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.005s
I20250623 08:04:28.526687 339 raft_consensus.cc:357] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.526950 339 raft_consensus.cc:383] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.527053 339 raft_consensus.cc:738] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.527324 339 consensus_queue.cc:260] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.528177 339 ts_tablet_manager.cc:1428] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250623 08:04:28.528544 339 tablet_bootstrap.cc:492] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.530227 699 raft_consensus.cc:491] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.530375 699 raft_consensus.cc:513] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.531035 699 leader_election.cc:290] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.532233 339 tablet_bootstrap.cc:654] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.532521 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9e53da264a5ca1f70e1846c906f0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.532723 32381 raft_consensus.cc:2466] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:28.533144 32453 leader_election.cc:304] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.533411 699 raft_consensus.cc:2802] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.533505 699 raft_consensus.cc:491] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.533597 699 raft_consensus.cc:3058] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.534585 342 tablet_bootstrap.cc:492] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.534679 339 tablet_bootstrap.cc:492] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.534832 342 ts_tablet_manager.cc:1397] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.017s
I20250623 08:04:28.534842 339 ts_tablet_manager.cc:1397] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20250623 08:04:28.535629 342 raft_consensus.cc:357] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.535614 339 raft_consensus.cc:357] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.535807 342 raft_consensus.cc:383] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.535800 339 raft_consensus.cc:383] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.535879 342 raft_consensus.cc:738] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.535956 339 raft_consensus.cc:738] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.536123 342 consensus_queue.cc:260] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.536185 339 consensus_queue.cc:260] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.536675 339 ts_tablet_manager.cc:1428] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:28.536947 339 tablet_bootstrap.cc:492] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.537962 342 ts_tablet_manager.cc:1428] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:28.538252 342 tablet_bootstrap.cc:492] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.539930 342 tablet_bootstrap.cc:654] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.542326 344 ts_tablet_manager.cc:1918] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:28.542482 344 log.cc:1199] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/16d6b1e81e4d438f85d8fee4dd7d8b7d
I20250623 08:04:28.542894 344 ts_tablet_manager.cc:1939] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:28.544909 339 tablet_bootstrap.cc:654] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.546396 344 tablet_replica.cc:331] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:28.546646 344 raft_consensus.cc:2241] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:28.546942 344 pending_rounds.cc:70] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:28.547010 344 pending_rounds.cc:77] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727393674186752 op_type: NO_OP noop_request { }
I20250623 08:04:28.547132 344 raft_consensus.cc:2887] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:28.547417 344 raft_consensus.cc:2270] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:28.547811 699 raft_consensus.cc:513] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.548656 699 leader_election.cc:290] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.549310 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9e53da264a5ca1f70e1846c906f0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.549506 32381 raft_consensus.cc:3058] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.550740 339 tablet_bootstrap.cc:492] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.550904 339 ts_tablet_manager.cc:1397] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20250623 08:04:28.551584 339 raft_consensus.cc:357] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.551703 339 raft_consensus.cc:383] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.551764 339 raft_consensus.cc:738] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.551947 339 consensus_queue.cc:260] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.552369 339 ts_tablet_manager.cc:1428] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:28.552520 32381 raft_consensus.cc:2466] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:28.552583 339 tablet_bootstrap.cc:492] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.553992 339 tablet_bootstrap.cc:654] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.556686 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9e53da264a5ca1f70e1846c906f0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.558368 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9e53da264a5ca1f70e1846c906f0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.559732 710 raft_consensus.cc:2747] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.549454 344 ts_tablet_manager.cc:1905] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:28.550307 342 tablet_bootstrap.cc:492] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.553217 32453 leader_election.cc:304] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:28.561407 32453 leader_election.cc:343] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:28.561826 32453 leader_election.cc:343] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.561905 343 tablet_bootstrap.cc:654] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.562393 339 tablet_bootstrap.cc:492] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.562558 339 ts_tablet_manager.cc:1397] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20250623 08:04:28.563355 339 raft_consensus.cc:357] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.563557 339 raft_consensus.cc:383] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.563621 339 raft_consensus.cc:738] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.563848 339 consensus_queue.cc:260] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.564051 699 raft_consensus.cc:2802] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.564491 339 ts_tablet_manager.cc:1428] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:28.564570 699 raft_consensus.cc:695] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:28.564961 339 tablet_bootstrap.cc:492] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.564904 699 consensus_queue.cc:237] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.566795 339 tablet_bootstrap.cc:654] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.567031 342 ts_tablet_manager.cc:1397] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.001s
I20250623 08:04:28.568064 342 raft_consensus.cc:357] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.569192 342 raft_consensus.cc:383] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.569464 342 raft_consensus.cc:738] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.569943 342 consensus_queue.cc:260] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.571270 342 ts_tablet_manager.cc:1428] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:28.572315 342 tablet_bootstrap.cc:492] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.574609 342 tablet_bootstrap.cc:654] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.568280 344 ts_tablet_manager.cc:1918] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:28.575337 344 log.cc:1199] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/d0ee5a4fe03848229524cc0105226d8c
I20250623 08:04:28.575762 343 tablet_bootstrap.cc:492] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.575840 344 ts_tablet_manager.cc:1939] T d0ee5a4fe03848229524cc0105226d8c P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:28.575944 343 ts_tablet_manager.cc:1397] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.005s
I20250623 08:04:28.576731 343 raft_consensus.cc:357] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.576938 343 raft_consensus.cc:383] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.577008 343 raft_consensus.cc:738] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.577385 343 consensus_queue.cc:260] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.580920 343 ts_tablet_manager.cc:1428] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250623 08:04:28.582438 343 tablet_bootstrap.cc:492] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.585647 339 tablet_bootstrap.cc:492] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.585804 339 ts_tablet_manager.cc:1397] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20250623 08:04:28.586565 339 raft_consensus.cc:357] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.586777 339 raft_consensus.cc:383] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.586870 339 raft_consensus.cc:738] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.587081 339 consensus_queue.cc:260] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.587585 339 ts_tablet_manager.cc:1428] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:28.587877 339 tablet_bootstrap.cc:492] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.589582 339 tablet_bootstrap.cc:654] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.596316 343 tablet_bootstrap.cc:654] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.596885 339 tablet_bootstrap.cc:492] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.597146 339 ts_tablet_manager.cc:1397] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20250623 08:04:28.598380 339 raft_consensus.cc:357] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.598735 339 raft_consensus.cc:383] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.598901 339 raft_consensus.cc:738] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.599272 339 consensus_queue.cc:260] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.600173 339 ts_tablet_manager.cc:1428] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:28.600715 339 tablet_bootstrap.cc:492] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.605679 641 consensus_queue.cc:1035] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:28.609926 339 tablet_bootstrap.cc:654] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.622606 343 tablet_bootstrap.cc:492] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.622802 343 ts_tablet_manager.cc:1397] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.003s
I20250623 08:04:28.623579 343 raft_consensus.cc:357] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.625213 343 raft_consensus.cc:383] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.625332 343 raft_consensus.cc:738] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.625595 343 consensus_queue.cc:260] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.626111 343 ts_tablet_manager.cc:1428] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:28.626796 343 tablet_bootstrap.cc:492] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.628505 343 tablet_bootstrap.cc:654] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.630338 339 tablet_bootstrap.cc:492] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.630551 339 ts_tablet_manager.cc:1397] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.006s
I20250623 08:04:28.630964 685 consensus_queue.cc:1035] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:28.631570 641 raft_consensus.cc:491] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.637976 641 raft_consensus.cc:513] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.638525 343 tablet_bootstrap.cc:492] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.638600 641 leader_election.cc:290] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.638684 343 ts_tablet_manager.cc:1397] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250623 08:04:28.639302 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35ab9f77a81a4490abb0d2c0c69cd90f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.639590 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35ab9f77a81a4490abb0d2c0c69cd90f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:28.640048 32314 leader_election.cc:343] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.640295 343 raft_consensus.cc:357] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:28.640444 32314 leader_election.cc:343] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.640475 343 raft_consensus.cc:383] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.640535 343 raft_consensus.cc:738] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.640542 32314 leader_election.cc:304] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:28.640760 343 consensus_queue.cc:260] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.641034 641 raft_consensus.cc:2747] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.641270 699 raft_consensus.cc:491] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.641417 699 raft_consensus.cc:513] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.641973 699 leader_election.cc:290] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.643225 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e22f9bae5f7248af99cec06ce6dbbd41" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:28.644752 32453 leader_election.cc:343] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.631420 339 raft_consensus.cc:357] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.645416 339 raft_consensus.cc:383] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.645390 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e22f9bae5f7248af99cec06ce6dbbd41" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.645624 339 raft_consensus.cc:738] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.645682 32381 raft_consensus.cc:2466] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:28.646064 343 ts_tablet_manager.cc:1428] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:28.646194 32453 leader_election.cc:304] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.646644 657 raft_consensus.cc:2802] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.646778 657 raft_consensus.cc:491] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.646862 657 raft_consensus.cc:3058] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.648173 343 tablet_bootstrap.cc:492] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.649715 657 raft_consensus.cc:513] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.650023 342 tablet_bootstrap.cc:492] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.650171 342 ts_tablet_manager.cc:1397] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.042s
I20250623 08:04:28.650329 657 leader_election.cc:290] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.650950 342 raft_consensus.cc:357] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.651151 342 raft_consensus.cc:383] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.651247 342 raft_consensus.cc:738] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.651394 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e22f9bae5f7248af99cec06ce6dbbd41" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.651489 342 consensus_queue.cc:260] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.651782 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e22f9bae5f7248af99cec06ce6dbbd41" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.651988 342 ts_tablet_manager.cc:1428] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:28.651985 32381 raft_consensus.cc:3058] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.652089 339 consensus_queue.cc:260] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.652288 342 tablet_bootstrap.cc:492] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.652868 339 ts_tablet_manager.cc:1428] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250623 08:04:28.653404 339 tablet_bootstrap.cc:492] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.653995 342 tablet_bootstrap.cc:654] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.655532 699 consensus_queue.cc:1035] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:28.657783 32453 leader_election.cc:343] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.658911 32381 raft_consensus.cc:2466] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:28.662684 32453 leader_election.cc:304] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.663190 699 raft_consensus.cc:2802] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.663347 699 raft_consensus.cc:695] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:28.663650 699 consensus_queue.cc:237] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.667258 342 tablet_bootstrap.cc:492] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.667403 342 ts_tablet_manager.cc:1397] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20250623 08:04:28.668112 342 raft_consensus.cc:357] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.668287 342 raft_consensus.cc:383] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.668390 342 raft_consensus.cc:738] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.668399 343 tablet_bootstrap.cc:654] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.668581 342 consensus_queue.cc:260] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.669179 342 ts_tablet_manager.cc:1428] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:28.669445 342 tablet_bootstrap.cc:492] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.674410 343 tablet_bootstrap.cc:492] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.674623 343 ts_tablet_manager.cc:1397] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20250623 08:04:28.674643 342 tablet_bootstrap.cc:654] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.675694 343 raft_consensus.cc:357] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.675875 343 raft_consensus.cc:383] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.675971 343 raft_consensus.cc:738] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.678356 343 consensus_queue.cc:260] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.687156 710 raft_consensus.cc:491] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.687314 710 raft_consensus.cc:513] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.687882 710 leader_election.cc:290] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:28.688503 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "966a159e212040109e2842c15b69fb71" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.688576 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "966a159e212040109e2842c15b69fb71" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.688759 32379 raft_consensus.cc:2466] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:28.688767 32518 raft_consensus.cc:2466] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:28.689121 32589 leader_election.cc:304] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.689523 710 raft_consensus.cc:2802] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.689630 710 raft_consensus.cc:491] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.689709 710 raft_consensus.cc:3058] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.692040 710 raft_consensus.cc:513] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.692835 342 tablet_bootstrap.cc:492] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.693037 342 ts_tablet_manager.cc:1397] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20250623 08:04:28.693198 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "966a159e212040109e2842c15b69fb71" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.693423 32379 raft_consensus.cc:3058] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.694037 342 raft_consensus.cc:357] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.694212 342 raft_consensus.cc:383] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.694267 342 raft_consensus.cc:738] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.694478 342 consensus_queue.cc:260] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.695091 342 ts_tablet_manager.cc:1428] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:28.696568 32379 raft_consensus.cc:2466] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:28.696580 710 leader_election.cc:290] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:28.696933 32589 leader_election.cc:304] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.697669 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "966a159e212040109e2842c15b69fb71" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.697788 342 tablet_bootstrap.cc:492] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:28.697746 32114 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 441607us
I20250623 08:04:28.697871 32520 raft_consensus.cc:3058] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.698194 32020 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskJxinPB/build/asan/bin/kudu with pid 32096
I20250623 08:04:28.699920 342 tablet_bootstrap.cc:654] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.701726 710 raft_consensus.cc:2802] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.701961 710 raft_consensus.cc:695] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:28.702225 710 consensus_queue.cc:237] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.705535 32520 raft_consensus.cc:2466] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:28.705842 339 tablet_bootstrap.cc:654] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.711354 343 ts_tablet_manager.cc:1428] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20250623 08:04:28.711867 343 tablet_bootstrap.cc:492] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.712972 339 tablet_bootstrap.cc:492] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.713131 339 ts_tablet_manager.cc:1397] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.000s
I20250623 08:04:28.713869 339 raft_consensus.cc:357] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.714077 339 raft_consensus.cc:383] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.714146 339 raft_consensus.cc:738] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.714360 339 consensus_queue.cc:260] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.715061 339 ts_tablet_manager.cc:1428] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:28.715634 342 tablet_bootstrap.cc:492] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.715777 342 ts_tablet_manager.cc:1397] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20250623 08:04:28.716162 343 tablet_bootstrap.cc:654] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.716516 342 raft_consensus.cc:357] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.716709 342 raft_consensus.cc:383] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.716810 342 raft_consensus.cc:738] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.717021 342 consensus_queue.cc:260] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.717656 342 ts_tablet_manager.cc:1428] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:28.718191 342 tablet_bootstrap.cc:492] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.719429 587 raft_consensus.cc:491] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.719584 587 raft_consensus.cc:513] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.719911 339 tablet_bootstrap.cc:492] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.719975 342 tablet_bootstrap.cc:654] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.720230 587 leader_election.cc:290] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.720971 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f02f36d6c1174a23a1f1043a431d0b6b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.721231 32379 raft_consensus.cc:2391] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:28.721561 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f02f36d6c1174a23a1f1043a431d0b6b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.726161 641 raft_consensus.cc:491] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.726326 641 raft_consensus.cc:513] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.726842 587 raft_consensus.cc:491] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.727005 587 raft_consensus.cc:513] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.727541 457 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 3 FOLLOWER]: Refusing update from remote peer d31835ec4ad74332a00998d431492466: Log matching property violated. Preceding OpId in replica: term: 2 index: 263. Preceding OpId from leader: term: 3 index: 357. (index mismatch)
I20250623 08:04:28.727609 587 leader_election.cc:290] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.728259 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ea81642050141a9be84f151a5785abc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.728763 641 leader_election.cc:290] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.729413 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "effd4fff43fe4d4fb36458986e1dfce1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.729468 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "effd4fff43fe4d4fb36458986e1dfce1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.729578 32518 raft_consensus.cc:2466] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
W20250623 08:04:28.730415 32314 leader_election.cc:343] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.730676 32314 leader_election.cc:304] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.730824 727 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 253, Time since last communication: 0.000s
I20250623 08:04:28.731169 641 raft_consensus.cc:2802] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.731300 641 raft_consensus.cc:491] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.731395 641 raft_consensus.cc:3058] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.735947 641 raft_consensus.cc:513] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.736862 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "effd4fff43fe4d4fb36458986e1dfce1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.737036 32518 raft_consensus.cc:3058] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.739034 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ea81642050141a9be84f151a5785abc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.739557 32589 leader_election.cc:304] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.739809 710 raft_consensus.cc:2747] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.739929 32518 raft_consensus.cc:2466] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.745502 343 tablet_bootstrap.cc:492] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.745684 343 ts_tablet_manager.cc:1397] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250623 08:04:28.746469 343 raft_consensus.cc:357] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.746667 343 raft_consensus.cc:383] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.746857 710 raft_consensus.cc:491] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.748293 343 raft_consensus.cc:738] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.748351 710 raft_consensus.cc:513] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.748394 32314 leader_election.cc:304] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.748567 343 consensus_queue.cc:260] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.748756 746 raft_consensus.cc:491] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.748986 710 leader_election.cc:290] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.748904 746 raft_consensus.cc:513] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.749074 343 ts_tablet_manager.cc:1428] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20250623 08:04:28.749356 343 tablet_bootstrap.cc:492] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.749650 746 leader_election.cc:290] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.749825 746 raft_consensus.cc:2802] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.749960 746 raft_consensus.cc:695] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:28.750141 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726d80e05a344f27b5a289723cb5aed7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.750229 746 consensus_queue.cc:237] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.752339 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0311ce67989f49e8866234a88e5b4d09" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.752599 32517 raft_consensus.cc:2466] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.752712 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "726d80e05a344f27b5a289723cb5aed7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.752928 32519 raft_consensus.cc:2391] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:28.752926 747 raft_consensus.cc:491] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.752976 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0311ce67989f49e8866234a88e5b4d09" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.753162 32656 raft_consensus.cc:2466] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.753044 747 raft_consensus.cc:513] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.753945 747 leader_election.cc:290] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.754804 32314 leader_election.cc:304] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.755026 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae7c2ac546114a819a543f1667fb7844" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.755218 32656 raft_consensus.cc:2466] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.755542 32589 leader_election.cc:304] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.755688 747 raft_consensus.cc:2802] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.755714 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae7c2ac546114a819a543f1667fb7844" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.755796 747 raft_consensus.cc:491] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.755878 747 raft_consensus.cc:3058] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.755913 32519 raft_consensus.cc:2466] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.755946 587 raft_consensus.cc:3058] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.756080 343 tablet_bootstrap.cc:654] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.757402 699 raft_consensus.cc:491] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.757534 699 raft_consensus.cc:513] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.758155 699 leader_election.cc:290] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.758896 747 raft_consensus.cc:513] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.759145 587 raft_consensus.cc:2747] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.759972 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e47b10f46b8402e94354ebca573c194" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:28.760454 32453 leader_election.cc:343] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.762038 342 tablet_bootstrap.cc:492] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.762203 342 ts_tablet_manager.cc:1397] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.016s
I20250623 08:04:28.762686 32314 leader_election.cc:304] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.762866 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0311ce67989f49e8866234a88e5b4d09" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.763031 685 raft_consensus.cc:2802] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.763057 32519 raft_consensus.cc:3058] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.763127 685 raft_consensus.cc:491] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.763401 685 raft_consensus.cc:3058] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.766083 32519 raft_consensus.cc:2466] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.766158 685 raft_consensus.cc:513] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.767118 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e47b10f46b8402e94354ebca573c194" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.767314 32379 raft_consensus.cc:2466] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:28.767601 342 raft_consensus.cc:357] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.767803 342 raft_consensus.cc:383] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.767887 342 raft_consensus.cc:738] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.766717 685 leader_election.cc:290] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.768133 342 consensus_queue.cc:260] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.768584 343 tablet_bootstrap.cc:492] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.768736 343 ts_tablet_manager.cc:1397] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20250623 08:04:28.768918 342 ts_tablet_manager.cc:1428] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250623 08:04:28.769340 342 tablet_bootstrap.cc:492] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.769562 343 raft_consensus.cc:357] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.769722 343 raft_consensus.cc:383] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.769976 747 leader_election.cc:290] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.769981 32589 leader_election.cc:304] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.770539 32314 leader_election.cc:304] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.771010 587 raft_consensus.cc:3058] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:28.771349 32314 consensus_peers.cc:487] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 228375592aee48a1aa21e1d16962b663. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:28.771405 342 tablet_bootstrap.cc:654] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.771862 32453 leader_election.cc:304] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.772009 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae7c2ac546114a819a543f1667fb7844" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.772203 32656 raft_consensus.cc:3058] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.772302 699 raft_consensus.cc:2802] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.772408 699 raft_consensus.cc:491] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.772503 699 raft_consensus.cc:3058] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.774120 587 raft_consensus.cc:2747] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.774673 343 raft_consensus.cc:738] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.775072 32656 raft_consensus.cc:2466] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.774995 343 consensus_queue.cc:260] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.775410 699 raft_consensus.cc:513] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.775805 343 ts_tablet_manager.cc:1428] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:28.776021 699 leader_election.cc:290] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.776214 343 tablet_bootstrap.cc:492] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.777869 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae7c2ac546114a819a543f1667fb7844" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.778143 343 tablet_bootstrap.cc:654] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.778131 32520 raft_consensus.cc:3058] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.783878 342 tablet_bootstrap.cc:492] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.784042 342 ts_tablet_manager.cc:1397] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20250623 08:04:28.784955 342 raft_consensus.cc:357] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.785137 342 raft_consensus.cc:383] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.785223 342 raft_consensus.cc:738] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.785193 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e47b10f46b8402e94354ebca573c194" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.785444 342 consensus_queue.cc:260] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.786070 342 ts_tablet_manager.cc:1428] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:28.786405 342 tablet_bootstrap.cc:492] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.788091 342 tablet_bootstrap.cc:654] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:28.786255 32453 leader_election.cc:343] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.790336 747 raft_consensus.cc:2802] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.790550 747 raft_consensus.cc:695] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:28.791041 343 tablet_bootstrap.cc:492] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.791213 343 ts_tablet_manager.cc:1397] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20250623 08:04:28.791589 641 leader_election.cc:290] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.792013 343 raft_consensus.cc:357] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.792212 343 raft_consensus.cc:383] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.792289 343 raft_consensus.cc:738] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.792544 343 consensus_queue.cc:260] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.793143 343 ts_tablet_manager.cc:1428] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:28.793471 343 tablet_bootstrap.cc:492] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.795445 343 tablet_bootstrap.cc:654] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.790938 747 consensus_queue.cc:237] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.796959 342 tablet_bootstrap.cc:492] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.797108 342 ts_tablet_manager.cc:1397] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250623 08:04:28.797256 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "effd4fff43fe4d4fb36458986e1dfce1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.797668 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0311ce67989f49e8866234a88e5b4d09" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.797852 32656 raft_consensus.cc:3058] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.797861 342 raft_consensus.cc:357] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.798056 342 raft_consensus.cc:383] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.798123 342 raft_consensus.cc:738] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.798378 342 consensus_queue.cc:260] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.799010 342 ts_tablet_manager.cc:1428] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250623 08:04:28.799311 342 tablet_bootstrap.cc:492] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.799376 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e47b10f46b8402e94354ebca573c194" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.799547 32379 raft_consensus.cc:3058] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.801110 342 tablet_bootstrap.cc:654] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.801137 32656 raft_consensus.cc:2466] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.802361 32379 raft_consensus.cc:2466] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:28.802886 32314 leader_election.cc:304] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:28.803244 32314 leader_election.cc:343] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.804018 32453 leader_election.cc:304] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.804407 699 raft_consensus.cc:2802] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.804527 699 raft_consensus.cc:695] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:28.804786 699 consensus_queue.cc:237] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.806175 746 raft_consensus.cc:2802] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.806303 746 raft_consensus.cc:695] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:28.806535 746 consensus_queue.cc:237] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.808014 685 raft_consensus.cc:491] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.808279 343 tablet_bootstrap.cc:492] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.808218 685 raft_consensus.cc:513] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.808446 343 ts_tablet_manager.cc:1397] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20250623 08:04:28.808990 685 leader_election.cc:290] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.809312 343 raft_consensus.cc:357] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.809475 343 raft_consensus.cc:383] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.809558 343 raft_consensus.cc:738] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.809784 343 consensus_queue.cc:260] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.811684 343 ts_tablet_manager.cc:1428] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:28.812007 343 tablet_bootstrap.cc:492] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.813818 343 tablet_bootstrap.cc:654] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.817196 342 tablet_bootstrap.cc:492] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.817348 342 ts_tablet_manager.cc:1397] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20250623 08:04:28.818223 342 raft_consensus.cc:357] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.818405 342 raft_consensus.cc:383] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.818500 342 raft_consensus.cc:738] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.818984 342 consensus_queue.cc:260] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.819859 342 ts_tablet_manager.cc:1428] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:28.819907 587 raft_consensus.cc:491] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.820012 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc95f12ab5d4b9b9f0729cb88aabbaa" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.820055 587 raft_consensus.cc:513] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.820204 342 tablet_bootstrap.cc:492] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.820765 587 leader_election.cc:290] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:28.821943 390 connection.cc:537] server connection from 127.0.0.1:48180 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250623 08:04:28.822263 342 tablet_bootstrap.cc:654] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:28.822501 32454 connection.cc:537] server connection from 127.0.0.1:55518 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:28.822649 32710 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.31.69.61:33313)
W20250623 08:04:28.823027 32710 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.69.61:33313)
I20250623 08:04:28.823035 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc95f12ab5d4b9b9f0729cb88aabbaa" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:28.823261 32710 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.31.69.61:33313)
I20250623 08:04:28.824090 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe30079045d437aa61bb5bfbfb96133" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:28.824143 32710 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.31.69.61:33313)
I20250623 08:04:28.824280 32379 raft_consensus.cc:2466] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:28.824606 32589 leader_election.cc:304] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.824851 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe30079045d437aa61bb5bfbfb96133" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.824888 587 raft_consensus.cc:2802] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.824998 587 raft_consensus.cc:491] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.825089 587 raft_consensus.cc:3058] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.825223 32518 raft_consensus.cc:2466] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
W20250623 08:04:28.826200 32314 leader_election.cc:343] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:28.826519 32314 leader_election.cc:343] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.826711 32314 leader_election.cc:304] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:28.827328 747 raft_consensus.cc:2747] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.827960 587 raft_consensus.cc:513] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.829460 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe30079045d437aa61bb5bfbfb96133" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.829537 587 leader_election.cc:290] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:28.829643 32379 raft_consensus.cc:3058] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.833091 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe30079045d437aa61bb5bfbfb96133" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.833308 32518 raft_consensus.cc:3058] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.836841 32518 raft_consensus.cc:2466] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:28.837445 32589 leader_election.cc:304] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:28.837819 587 raft_consensus.cc:2802] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.837970 587 raft_consensus.cc:695] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:28.838366 32379 raft_consensus.cc:2466] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:28.838320 587 consensus_queue.cc:237] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.823287 32520 raft_consensus.cc:2466] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.841871 339 tablet_bootstrap.cc:654] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.846818 699 raft_consensus.cc:491] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.847046 699 raft_consensus.cc:513] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.848256 699 leader_election.cc:290] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.848662 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35ab9f77a81a4490abb0d2c0c69cd90f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.848922 32656 raft_consensus.cc:2466] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:28.849565 32453 leader_election.cc:304] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:28.850878 699 raft_consensus.cc:2802] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.851002 699 raft_consensus.cc:491] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.851125 699 raft_consensus.cc:3058] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.854179 699 raft_consensus.cc:513] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.855172 699 leader_election.cc:290] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.858180 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35ab9f77a81a4490abb0d2c0c69cd90f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.858413 32381 raft_consensus.cc:3058] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:28.859035 32455 connection.cc:537] client connection to 127.31.69.61:33313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:28.859485 32590 connection.cc:537] server connection from 127.0.0.1:36600 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250623 08:04:28.862270 32381 raft_consensus.cc:2466] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:28.862972 32045 connection.cc:537] client connection to 127.31.69.61:33313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:28.863101 32045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 8 similar messages]
I20250623 08:04:28.878336 343 tablet_bootstrap.cc:492] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.878561 343 ts_tablet_manager.cc:1397] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.015s
I20250623 08:04:28.879573 343 raft_consensus.cc:357] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.879794 343 raft_consensus.cc:383] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.879885 343 raft_consensus.cc:738] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.880125 343 consensus_queue.cc:260] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.882827 343 ts_tablet_manager.cc:1428] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250623 08:04:28.883153 343 tablet_bootstrap.cc:492] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.883126 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35ab9f77a81a4490abb0d2c0c69cd90f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.883467 32379 raft_consensus.cc:2374] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1. Re-sending same reply.
W20250623 08:04:28.886382 798 negotiation.cc:337] Failed RPC negotiation. Trace:
0623 08:04:28.825907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.69.61:33313 (local address 127.0.0.1:48190)
0623 08:04:28.884294 (+ 58387us) negotiation.cc:107] Waiting for socket to connect
0623 08:04:28.884452 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":58185,"threads_started":1}
I20250623 08:04:28.886834 343 tablet_bootstrap.cc:654] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.887449 32453 leader_election.cc:304] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.888540 699 raft_consensus.cc:2802] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.888851 699 raft_consensus.cc:695] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:28.889155 699 consensus_queue.cc:237] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:28.892427 803 negotiation.cc:337] Failed RPC negotiation. Trace:
0623 08:04:28.853502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.69.61:33313 (local address 127.31.69.3:57643)
0623 08:04:28.892058 (+ 38556us) negotiation.cc:107] Waiting for socket to connect
0623 08:04:28.892188 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38442,"threads_started":1}
I20250623 08:04:28.893500 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35ab9f77a81a4490abb0d2c0c69cd90f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.893720 32656 raft_consensus.cc:3058] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.896828 32656 raft_consensus.cc:2466] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:28.902799 339 tablet_bootstrap.cc:492] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.903018 339 ts_tablet_manager.cc:1397] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.183s user 0.002s sys 0.002s
I20250623 08:04:28.905282 339 raft_consensus.cc:357] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.905936 339 raft_consensus.cc:383] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.906087 339 raft_consensus.cc:738] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.906370 339 consensus_queue.cc:260] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.907137 339 ts_tablet_manager.cc:1428] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:28.907704 339 tablet_bootstrap.cc:492] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.909550 339 tablet_bootstrap.cc:654] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:28.909598 32045 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.61:33313, user_credentials={real_user=slave}} blocked reactor thread for 46462.4us
I20250623 08:04:28.911317 343 tablet_bootstrap.cc:492] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.911484 343 ts_tablet_manager.cc:1397] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.004s
I20250623 08:04:28.912238 343 raft_consensus.cc:357] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.912422 343 raft_consensus.cc:383] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.912524 343 raft_consensus.cc:738] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.912744 343 consensus_queue.cc:260] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.913502 343 ts_tablet_manager.cc:1428] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:28.913862 343 tablet_bootstrap.cc:492] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.915598 343 tablet_bootstrap.cc:654] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.916332 342 tablet_bootstrap.cc:492] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.916569 342 ts_tablet_manager.cc:1397] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.024s
I20250623 08:04:28.917412 342 raft_consensus.cc:357] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.917697 342 raft_consensus.cc:383] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.917829 342 raft_consensus.cc:738] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.918150 342 consensus_queue.cc:260] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.918767 342 ts_tablet_manager.cc:1428] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:28.919296 342 tablet_bootstrap.cc:492] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.921061 342 tablet_bootstrap.cc:654] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.927484 747 raft_consensus.cc:491] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.927680 747 raft_consensus.cc:513] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.928491 747 leader_election.cc:290] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.929271 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b980c746a3c042bfae90aa076acbb1df" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.929461 32520 raft_consensus.cc:2466] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.929533 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b980c746a3c042bfae90aa076acbb1df" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:28.929772 32656 raft_consensus.cc:2466] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.930282 32314 leader_election.cc:304] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.930853 747 raft_consensus.cc:2802] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.931015 747 raft_consensus.cc:491] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.931128 747 raft_consensus.cc:3058] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:28.931710 32045 consensus_peers.cc:487] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 -> Peer 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313): Couldn't send request to peer 9752155c20ee42fbb7ff7b7d5b682444. Status: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:28.934792 747 raft_consensus.cc:513] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.935578 747 leader_election.cc:290] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.936589 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b980c746a3c042bfae90aa076acbb1df" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.936780 32656 raft_consensus.cc:3058] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.936806 339 tablet_bootstrap.cc:492] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.936970 339 ts_tablet_manager.cc:1397] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.009s
I20250623 08:04:28.937755 339 raft_consensus.cc:357] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.937944 339 raft_consensus.cc:383] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.938017 339 raft_consensus.cc:738] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.938242 339 consensus_queue.cc:260] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.938537 342 tablet_bootstrap.cc:492] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.938680 342 ts_tablet_manager.cc:1397] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20250623 08:04:28.938969 339 ts_tablet_manager.cc:1428] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:28.939424 339 tablet_bootstrap.cc:492] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.939460 342 raft_consensus.cc:357] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.939640 342 raft_consensus.cc:383] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.939730 342 raft_consensus.cc:738] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.939930 342 consensus_queue.cc:260] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.940584 342 ts_tablet_manager.cc:1428] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:28.940876 342 tablet_bootstrap.cc:492] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.940958 587 raft_consensus.cc:491] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.941110 587 raft_consensus.cc:513] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.941722 587 leader_election.cc:290] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.942480 587 raft_consensus.cc:491] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.942631 342 tablet_bootstrap.cc:654] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.942621 587 raft_consensus.cc:513] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.942680 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a97b34e386443fa19a9601621b5017" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.942896 32379 raft_consensus.cc:2466] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:28.943195 587 leader_election.cc:290] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.943240 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a97b34e386443fa19a9601621b5017" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.943579 339 tablet_bootstrap.cc:654] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.943874 32518 raft_consensus.cc:2466] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:28.943933 32589 leader_election.cc:304] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.944169 587 raft_consensus.cc:2802] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.944228 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "776b412b3f564a7581c6a5e1c84e8499" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:28.944273 587 raft_consensus.cc:491] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.944362 587 raft_consensus.cc:3058] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.944437 32379 raft_consensus.cc:2391] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:28.945554 32656 raft_consensus.cc:2466] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.946455 32314 leader_election.cc:304] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.947371 641 raft_consensus.cc:2802] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.947511 641 raft_consensus.cc:695] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:28.947778 641 consensus_queue.cc:237] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.947953 587 raft_consensus.cc:513] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.948486 587 leader_election.cc:290] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:28.948693 747 raft_consensus.cc:491] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:28.948822 747 raft_consensus.cc:513] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.949738 747 leader_election.cc:290] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.950863 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7da84969a6464b5caeaf47af0cbaaa4d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:28.951330 32314 leader_election.cc:343] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.952842 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a97b34e386443fa19a9601621b5017" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:28.953034 32379 raft_consensus.cc:3058] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.955883 32379 raft_consensus.cc:2466] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:28.956506 32589 leader_election.cc:304] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:28.956805 587 raft_consensus.cc:2802] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.956946 587 raft_consensus.cc:695] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:28.957616 587 consensus_queue.cc:237] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.958324 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7da84969a6464b5caeaf47af0cbaaa4d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.959803 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "776b412b3f564a7581c6a5e1c84e8499" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:28.961445 343 tablet_bootstrap.cc:492] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.961619 343 ts_tablet_manager.cc:1397] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.013s
I20250623 08:04:28.962468 343 raft_consensus.cc:357] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.962659 343 raft_consensus.cc:383] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.962785 343 raft_consensus.cc:738] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.963248 32589 leader_election.cc:304] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:28.963712 710 raft_consensus.cc:3058] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.965050 343 consensus_queue.cc:260] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:28.966926 32518 raft_consensus.cc:2466] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:28.967202 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b980c746a3c042bfae90aa076acbb1df" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.967401 32520 raft_consensus.cc:3058] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.970995 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6a97b34e386443fa19a9601621b5017" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.971158 32517 raft_consensus.cc:3058] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.974016 343 ts_tablet_manager.cc:1428] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20250623 08:04:28.974037 32517 raft_consensus.cc:2466] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:28.970383 32520 raft_consensus.cc:2466] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.974558 342 tablet_bootstrap.cc:492] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:28.974736 342 ts_tablet_manager.cc:1397] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.007s
I20250623 08:04:28.975174 343 tablet_bootstrap.cc:492] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:28.975401 32314 leader_election.cc:304] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.976403 641 raft_consensus.cc:2802] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:28.976524 641 raft_consensus.cc:491] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:28.976616 641 raft_consensus.cc:3058] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.977146 343 tablet_bootstrap.cc:654] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.979449 339 tablet_bootstrap.cc:492] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:28.979607 339 ts_tablet_manager.cc:1397] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.004s
I20250623 08:04:28.979629 710 raft_consensus.cc:2747] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:28.979460 342 raft_consensus.cc:357] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.979835 342 raft_consensus.cc:383] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.979755 641 raft_consensus.cc:513] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.979909 342 raft_consensus.cc:738] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.980201 342 consensus_queue.cc:260] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.980470 339 raft_consensus.cc:357] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.980760 342 ts_tablet_manager.cc:1428] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250623 08:04:28.980826 339 raft_consensus.cc:383] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.980855 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7da84969a6464b5caeaf47af0cbaaa4d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:28.980944 339 raft_consensus.cc:738] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.981078 32520 raft_consensus.cc:3058] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:28.981168 339 consensus_queue.cc:260] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:28.984899 339 ts_tablet_manager.cc:1428] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:28.985165 339 tablet_bootstrap.cc:492] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:28.988389 342 tablet_bootstrap.cc:492] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:28.990101 342 tablet_bootstrap.cc:654] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.990401 343 tablet_bootstrap.cc:492] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:28.990558 343 ts_tablet_manager.cc:1397] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20250623 08:04:28.991434 343 raft_consensus.cc:357] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.991647 343 raft_consensus.cc:383] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:28.991694 339 tablet_bootstrap.cc:654] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:28.991729 343 raft_consensus.cc:738] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:28.993814 343 consensus_queue.cc:260] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:28.994733 641 leader_election.cc:290] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:28.995183 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7da84969a6464b5caeaf47af0cbaaa4d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:28.996382 343 ts_tablet_manager.cc:1428] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20250623 08:04:28.996524 32314 leader_election.cc:343] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:28.997082 32520 raft_consensus.cc:2466] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:28.997507 32314 leader_election.cc:304] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:28.997951 641 raft_consensus.cc:2802] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:28.998401 641 raft_consensus.cc:695] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:28.998730 641 consensus_queue.cc:237] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.001969 343 tablet_bootstrap.cc:492] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.005482 339 tablet_bootstrap.cc:492] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.005686 339 ts_tablet_manager.cc:1397] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20250623 08:04:29.005914 343 tablet_bootstrap.cc:654] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.006466 339 raft_consensus.cc:357] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.006701 339 raft_consensus.cc:383] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.006819 339 raft_consensus.cc:738] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.007093 339 consensus_queue.cc:260] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.007714 339 ts_tablet_manager.cc:1428] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250623 08:04:29.009505 343 tablet_bootstrap.cc:492] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.009676 343 ts_tablet_manager.cc:1397] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20250623 08:04:29.010914 339 tablet_bootstrap.cc:492] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.012097 343 raft_consensus.cc:357] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.012277 343 raft_consensus.cc:383] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.012342 343 raft_consensus.cc:738] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.012758 343 consensus_queue.cc:260] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.013530 343 ts_tablet_manager.cc:1428] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:29.015571 343 tablet_bootstrap.cc:492] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.016696 339 tablet_bootstrap.cc:654] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.017741 343 tablet_bootstrap.cc:654] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.018728 342 tablet_bootstrap.cc:492] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.018919 342 ts_tablet_manager.cc:1397] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20250623 08:04:29.019773 342 raft_consensus.cc:357] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.019958 342 raft_consensus.cc:383] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.020061 342 raft_consensus.cc:738] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.020287 342 consensus_queue.cc:260] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.020998 342 ts_tablet_manager.cc:1428] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:29.021597 342 tablet_bootstrap.cc:492] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.025223 342 tablet_bootstrap.cc:654] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.034227 699 raft_consensus.cc:491] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.034436 699 raft_consensus.cc:513] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.035296 699 leader_election.cc:290] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.036594 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861107ab4624a05bd7a4d24b420c502" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.036726 339 tablet_bootstrap.cc:492] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.036828 32379 raft_consensus.cc:2466] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:29.036871 339 ts_tablet_manager.cc:1397] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.005s
I20250623 08:04:29.037226 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861107ab4624a05bd7a4d24b420c502" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.037452 32656 raft_consensus.cc:2466] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:29.037763 339 raft_consensus.cc:357] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.037922 339 raft_consensus.cc:383] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.037998 339 raft_consensus.cc:738] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.038277 339 consensus_queue.cc:260] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.038628 32453 leader_election.cc:304] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.039168 699 raft_consensus.cc:2802] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.039285 699 raft_consensus.cc:491] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.039384 699 raft_consensus.cc:3058] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.041167 339 ts_tablet_manager.cc:1428] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:29.042070 339 tablet_bootstrap.cc:492] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.042212 699 raft_consensus.cc:513] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.043736 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861107ab4624a05bd7a4d24b420c502" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.043898 32379 raft_consensus.cc:3058] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.045352 339 tablet_bootstrap.cc:654] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.048552 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861107ab4624a05bd7a4d24b420c502" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.048780 32656 raft_consensus.cc:3058] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.051091 699 leader_election.cc:290] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.052232 32656 raft_consensus.cc:2466] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:29.052846 32453 leader_election.cc:304] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.053331 641 consensus_queue.cc:1035] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.053651 32379 raft_consensus.cc:2466] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:29.054049 699 raft_consensus.cc:2802] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.056201 699 raft_consensus.cc:695] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:29.056644 699 consensus_queue.cc:237] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.077720 587 raft_consensus.cc:491] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.077913 587 raft_consensus.cc:513] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.078508 587 leader_election.cc:290] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.078954 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0117cbb09dd4067836266c9d6a32adf" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.079099 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0117cbb09dd4067836266c9d6a32adf" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.079345 32379 raft_consensus.cc:2466] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:29.079370 32520 raft_consensus.cc:2466] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:29.079876 32589 leader_election.cc:304] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.080202 587 raft_consensus.cc:2802] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.080287 587 raft_consensus.cc:491] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.080380 587 raft_consensus.cc:3058] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.083668 587 raft_consensus.cc:513] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.084241 587 leader_election.cc:290] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.084931 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0117cbb09dd4067836266c9d6a32adf" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.085115 32520 raft_consensus.cc:3058] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.088267 32520 raft_consensus.cc:2466] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:29.088821 587 raft_consensus.cc:491] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.088851 32589 leader_election.cc:304] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.088995 587 raft_consensus.cc:513] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.089118 710 raft_consensus.cc:2802] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.089395 710 raft_consensus.cc:695] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:29.089668 587 leader_election.cc:290] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.089697 710 consensus_queue.cc:237] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.091945 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e4678ca1c8f498c83c6bc2553882120" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.092190 32381 raft_consensus.cc:2466] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:29.092926 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e4678ca1c8f498c83c6bc2553882120" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.093161 32517 raft_consensus.cc:2466] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:29.095227 32589 leader_election.cc:304] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.095727 819 raft_consensus.cc:2802] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.095844 819 raft_consensus.cc:491] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.095964 819 raft_consensus.cc:3058] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.099205 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0117cbb09dd4067836266c9d6a32adf" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.099448 32379 raft_consensus.cc:3058] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:29.100965 32589 consensus_peers.cc:487] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 696f5d2797ef463bb72a2119501c6ef3. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:29.100914 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d31835ec4ad74332a00998d431492466. Latest consensus state: current_term: 3 leader_uuid: "d31835ec4ad74332a00998d431492466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:29.101184 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:29.101370 819 raft_consensus.cc:513] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.102130 819 leader_election.cc:290] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.102969 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e4678ca1c8f498c83c6bc2553882120" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.103178 32381 raft_consensus.cc:3058] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.106115 32381 raft_consensus.cc:2466] T 5e4678ca1c8f498c83c6bc2553882120 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:29.111496 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e4678ca1c8f498c83c6bc2553882120" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.111730 32520 raft_consensus.cc:3058] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.113917 32589 leader_election.cc:304] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.114626 710 raft_consensus.cc:491] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.114776 710 raft_consensus.cc:513] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.115374 710 leader_election.cc:290] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.115522 710 raft_consensus.cc:2802] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.115648 710 raft_consensus.cc:695] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:29.115911 710 consensus_queue.cc:237] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.116829 32520 raft_consensus.cc:2466] T 5e4678ca1c8f498c83c6bc2553882120 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:29.114367 32379 raft_consensus.cc:2466] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:29.118013 342 tablet_bootstrap.cc:492] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.118193 342 ts_tablet_manager.cc:1397] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.097s user 0.013s sys 0.006s
I20250623 08:04:29.119087 342 raft_consensus.cc:357] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.119408 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9918b15e48144ef0b8f3aedc2135be59" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.119621 32520 raft_consensus.cc:2391] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:29.119778 342 raft_consensus.cc:383] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.119866 342 raft_consensus.cc:738] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.120098 342 consensus_queue.cc:260] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.122231 342 ts_tablet_manager.cc:1428] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250623 08:04:29.122617 342 tablet_bootstrap.cc:492] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.124276 342 tablet_bootstrap.cc:654] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.125837 343 tablet_bootstrap.cc:492] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.125981 343 ts_tablet_manager.cc:1397] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.111s user 0.024s sys 0.049s
I20250623 08:04:29.126706 343 raft_consensus.cc:357] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:29.126899 343 raft_consensus.cc:383] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.126924 339 tablet_bootstrap.cc:492] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.126981 343 raft_consensus.cc:738] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.127095 339 ts_tablet_manager.cc:1397] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.010s
I20250623 08:04:29.127825 339 raft_consensus.cc:357] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.128034 339 raft_consensus.cc:383] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.128111 339 raft_consensus.cc:738] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.128357 339 consensus_queue.cc:260] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.129130 32656 raft_consensus.cc:3058] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.129205 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9918b15e48144ef0b8f3aedc2135be59" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.132735 32589 leader_election.cc:304] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:29.136680 343 consensus_queue.cc:260] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:29.137300 343 ts_tablet_manager.cc:1428] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250623 08:04:29.137480 710 raft_consensus.cc:2747] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:29.139923 343 tablet_bootstrap.cc:492] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.147819 343 tablet_bootstrap.cc:654] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.149467 339 ts_tablet_manager.cc:1428] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:29.150439 746 consensus_queue.cc:1035] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 08:04:29.151841 610 raft_consensus.cc:491] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.151998 610 raft_consensus.cc:513] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.153651 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485538f7c63948818f20a25c02381519" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.153894 32520 raft_consensus.cc:2466] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:29.154451 610 leader_election.cc:290] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.154911 819 raft_consensus.cc:491] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.155069 819 raft_consensus.cc:513] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:29.155642 819 leader_election.cc:290] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.155893 32314 leader_election.cc:304] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.156143 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485538f7c63948818f20a25c02381519" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.156451 746 raft_consensus.cc:2802] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.156544 746 raft_consensus.cc:491] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.156644 746 raft_consensus.cc:3058] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.159302 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5629d11c44e241d598f3e519b2fe1a44" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.159539 32520 raft_consensus.cc:2466] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
W20250623 08:04:29.159951 32314 leader_election.cc:343] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.159941 32589 leader_election.cc:304] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.160234 819 raft_consensus.cc:2802] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.160321 819 raft_consensus.cc:491] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.160430 819 raft_consensus.cc:3058] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.161165 746 raft_consensus.cc:513] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.161837 746 leader_election.cc:290] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.162639 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485538f7c63948818f20a25c02381519" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.162842 32520 raft_consensus.cc:3058] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.163166 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485538f7c63948818f20a25c02381519" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.163653 819 raft_consensus.cc:513] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:29.164247 819 leader_election.cc:290] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.164938 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5629d11c44e241d598f3e519b2fe1a44" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
W20250623 08:04:29.165125 32314 leader_election.cc:343] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.165181 32517 raft_consensus.cc:3058] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.167235 32520 raft_consensus.cc:2466] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.167690 32314 leader_election.cc:304] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:29.168486 746 raft_consensus.cc:2802] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.168838 32517 raft_consensus.cc:2466] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:29.169780 746 raft_consensus.cc:695] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:29.169889 343 tablet_bootstrap.cc:492] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.170068 343 ts_tablet_manager.cc:1397] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.003s
I20250623 08:04:29.170096 746 consensus_queue.cc:237] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.170336 342 tablet_bootstrap.cc:492] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.170512 342 ts_tablet_manager.cc:1397] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.019s
I20250623 08:04:29.170809 32589 leader_election.cc:304] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.171726 342 raft_consensus.cc:357] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.171964 342 raft_consensus.cc:383] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.172009 343 raft_consensus.cc:357] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.172120 342 raft_consensus.cc:738] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.172214 343 raft_consensus.cc:383] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.172271 819 raft_consensus.cc:2802] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.172312 343 raft_consensus.cc:738] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.172441 819 raft_consensus.cc:695] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:29.172642 342 consensus_queue.cc:260] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.172878 819 consensus_queue.cc:237] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:29.176921 343 consensus_queue.cc:260] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.178146 342 ts_tablet_manager.cc:1428] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250623 08:04:29.179111 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5629d11c44e241d598f3e519b2fe1a44" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.179348 32379 raft_consensus.cc:2466] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:29.180078 343 ts_tablet_manager.cc:1428] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250623 08:04:29.180804 342 tablet_bootstrap.cc:492] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.181119 343 tablet_bootstrap.cc:492] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.182650 342 tablet_bootstrap.cc:654] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.182873 343 tablet_bootstrap.cc:654] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.179114 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5629d11c44e241d598f3e519b2fe1a44" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.183238 32381 raft_consensus.cc:3058] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.185992 32381 raft_consensus.cc:2466] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:29.187413 339 tablet_bootstrap.cc:492] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.193240 339 tablet_bootstrap.cc:654] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.201014 746 consensus_queue.cc:1035] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.204376 710 raft_consensus.cc:491] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.204646 710 raft_consensus.cc:513] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.205513 710 leader_election.cc:290] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.206825 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e22f9bae5f7248af99cec06ce6dbbd41" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.206826 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e22f9bae5f7248af99cec06ce6dbbd41" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.207057 32381 raft_consensus.cc:2391] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:29.208150 32589 leader_election.cc:304] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:29.208913 710 raft_consensus.cc:3058] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.211688 699 raft_consensus.cc:491] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.211917 699 raft_consensus.cc:513] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.212726 699 leader_election.cc:290] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.214103 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3c850021e914530bb106ab7166be7c9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.214370 32656 raft_consensus.cc:2466] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:29.215184 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3c850021e914530bb106ab7166be7c9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.216965 32453 leader_election.cc:304] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.217447 699 raft_consensus.cc:2802] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.217568 699 raft_consensus.cc:491] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.217695 699 raft_consensus.cc:3058] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:29.219306 32453 leader_election.cc:343] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.220261 610 consensus_queue.cc:1035] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.223109 657 raft_consensus.cc:491] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.223258 657 raft_consensus.cc:513] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.223953 657 leader_election.cc:290] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.225150 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26f4a1b472e848ccb9a1ec71c1dd3516" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.227083 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26f4a1b472e848ccb9a1ec71c1dd3516" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.227324 32381 raft_consensus.cc:2466] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:29.227800 32453 leader_election.cc:304] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.228093 657 raft_consensus.cc:2802] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250623 08:04:29.228150 32453 leader_election.cc:343] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.228209 657 raft_consensus.cc:491] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.228291 657 raft_consensus.cc:3058] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.229707 710 raft_consensus.cc:2747] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:29.231098 657 raft_consensus.cc:513] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.232560 339 tablet_bootstrap.cc:492] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.232729 339 ts_tablet_manager.cc:1397] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.003s
I20250623 08:04:29.232163 699 raft_consensus.cc:513] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.233599 339 raft_consensus.cc:357] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.233798 339 raft_consensus.cc:383] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.233882 339 raft_consensus.cc:738] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.234089 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26f4a1b472e848ccb9a1ec71c1dd3516" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.235610 746 consensus_queue.cc:1035] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.235865 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26f4a1b472e848ccb9a1ec71c1dd3516" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.236088 32381 raft_consensus.cc:3058] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.239928 32381 raft_consensus.cc:2466] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:29.233515 657 leader_election.cc:290] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.241276 339 consensus_queue.cc:260] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:29.234918 32453 leader_election.cc:343] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.242630 32453 leader_election.cc:304] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:29.243299 657 raft_consensus.cc:2802] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.243461 657 raft_consensus.cc:695] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:29.243705 339 ts_tablet_manager.cc:1428] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250623 08:04:29.243796 657 consensus_queue.cc:237] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.247398 339 tablet_bootstrap.cc:492] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.250029 699 leader_election.cc:290] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.251936 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3c850021e914530bb106ab7166be7c9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.252338 32654 raft_consensus.cc:3058] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.254722 339 tablet_bootstrap.cc:654] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.255007 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3c850021e914530bb106ab7166be7c9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.255460 32381 raft_consensus.cc:3058] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.257004 699 consensus_queue.cc:1035] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.264111 848 consensus_queue.cc:1035] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.269035 710 raft_consensus.cc:491] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.269224 710 raft_consensus.cc:513] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:29.269937 710 leader_election.cc:290] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.270702 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9e53da264a5ca1f70e1846c906f0" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.270985 32379 raft_consensus.cc:2391] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:29.274224 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9e53da264a5ca1f70e1846c906f0" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.275156 832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d31835ec4ad74332a00998d431492466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:29.275480 832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: This master's current role is: LEADER
I20250623 08:04:29.275557 32589 leader_election.cc:304] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:29.275974 710 raft_consensus.cc:3058] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.276911 339 tablet_bootstrap.cc:492] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.276908 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d31835ec4ad74332a00998d431492466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:29.277110 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:29.277217 339 ts_tablet_manager.cc:1397] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.014s
I20250623 08:04:29.277963 339 raft_consensus.cc:357] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.278204 339 raft_consensus.cc:383] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.278301 339 raft_consensus.cc:738] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.279480 342 tablet_bootstrap.cc:492] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.279675 342 ts_tablet_manager.cc:1397] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.099s user 0.007s sys 0.045s
I20250623 08:04:29.280403 710 raft_consensus.cc:2747] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:29.280869 32381 raft_consensus.cc:2466] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:29.280434 343 tablet_bootstrap.cc:492] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.281256 343 ts_tablet_manager.cc:1397] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.018s
I20250623 08:04:29.281975 343 raft_consensus.cc:357] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.282130 343 raft_consensus.cc:383] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.282198 343 raft_consensus.cc:738] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.282424 343 consensus_queue.cc:260] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.282968 343 ts_tablet_manager.cc:1428] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:29.283222 343 tablet_bootstrap.cc:492] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.280970 339 consensus_queue.cc:260] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.284299 339 ts_tablet_manager.cc:1428] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250623 08:04:29.284770 342 raft_consensus.cc:357] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.284965 342 raft_consensus.cc:383] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.285032 342 raft_consensus.cc:738] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.285276 342 consensus_queue.cc:260] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.285799 342 ts_tablet_manager.cc:1428] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:29.286062 343 tablet_bootstrap.cc:654] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.287139 339 tablet_bootstrap.cc:492] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.288938 339 tablet_bootstrap.cc:654] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.290928 342 tablet_bootstrap.cc:492] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:29.291584 32589 consensus_peers.cc:487] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 696f5d2797ef463bb72a2119501c6ef3. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:29.295609 342 tablet_bootstrap.cc:654] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.299710 32453 leader_election.cc:304] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.300642 849 raft_consensus.cc:2802] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.302407 849 raft_consensus.cc:695] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:29.301811 699 consensus_queue.cc:1035] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 08:04:29.303380 849 consensus_queue.cc:237] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.311282 32654 raft_consensus.cc:2466] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:29.319372 343 tablet_bootstrap.cc:492] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.319587 343 ts_tablet_manager.cc:1397] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.008s
I20250623 08:04:29.320377 343 raft_consensus.cc:357] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.320590 343 raft_consensus.cc:383] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.320650 343 raft_consensus.cc:738] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.320869 343 consensus_queue.cc:260] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.321354 343 ts_tablet_manager.cc:1428] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:29.321863 343 tablet_bootstrap.cc:492] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.323705 343 tablet_bootstrap.cc:654] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.332070 710 raft_consensus.cc:491] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.332302 710 raft_consensus.cc:513] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.333037 710 leader_election.cc:290] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.334327 343 tablet_bootstrap.cc:492] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.334481 343 ts_tablet_manager.cc:1397] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20250623 08:04:29.335433 343 raft_consensus.cc:357] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.335629 343 raft_consensus.cc:383] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.335726 343 raft_consensus.cc:738] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.335918 343 consensus_queue.cc:260] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.337881 343 ts_tablet_manager.cc:1428] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:29.338181 343 tablet_bootstrap.cc:492] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.339987 343 tablet_bootstrap.cc:654] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.340260 731 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 08:04:29.345575 343 tablet_bootstrap.cc:492] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.345755 343 ts_tablet_manager.cc:1397] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250623 08:04:29.346553 343 raft_consensus.cc:357] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.348278 343 raft_consensus.cc:383] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.348371 343 raft_consensus.cc:738] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.348647 343 consensus_queue.cc:260] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.351437 343 ts_tablet_manager.cc:1428] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:29.351794 343 tablet_bootstrap.cc:492] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.352332 731 catalog_manager.cc:671] Loaded metadata for table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]
I20250623 08:04:29.353027 731 catalog_manager.cc:671] Loaded metadata for table default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]
I20250623 08:04:29.353596 731 catalog_manager.cc:671] Loaded metadata for table default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]
I20250623 08:04:29.353621 343 tablet_bootstrap.cc:654] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.355691 731 catalog_manager.cc:671] Loaded metadata for table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]
I20250623 08:04:29.358989 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc95f12ab5d4b9b9f0729cb88aabbaa" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.359216 32518 raft_consensus.cc:2466] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:29.359496 848 raft_consensus.cc:491] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.359632 848 raft_consensus.cc:513] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.360311 848 leader_election.cc:290] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.361519 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "175ee90133994154b7e4b58b0770db3b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.361812 32380 raft_consensus.cc:2466] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:29.362327 32589 leader_election.cc:304] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.362695 710 raft_consensus.cc:2802] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.362819 710 raft_consensus.cc:491] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.362926 710 raft_consensus.cc:3058] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.362920 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc95f12ab5d4b9b9f0729cb88aabbaa" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.363113 32381 raft_consensus.cc:2466] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:29.364342 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "175ee90133994154b7e4b58b0770db3b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.364616 32656 raft_consensus.cc:2466] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:29.365016 32453 leader_election.cc:304] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.365536 657 raft_consensus.cc:2802] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.365662 657 raft_consensus.cc:491] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.365753 657 raft_consensus.cc:3058] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.365933 710 raft_consensus.cc:513] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.366524 710 leader_election.cc:290] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.368626 657 raft_consensus.cc:513] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.369340 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc95f12ab5d4b9b9f0729cb88aabbaa" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.369539 32518 raft_consensus.cc:3058] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.369724 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc95f12ab5d4b9b9f0729cb88aabbaa" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.369925 32379 raft_consensus.cc:3058] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.370281 699 consensus_queue.cc:1035] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.370816 699 consensus_queue.cc:1035] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.373134 32518 raft_consensus.cc:2466] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:29.373708 657 leader_election.cc:290] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.376296 32379 raft_consensus.cc:2466] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:29.376792 32589 leader_election.cc:304] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.377035 731 catalog_manager.cc:671] Loaded metadata for table default.Table_6bb7361aabf7440589a3958da502f36c [id=150ddf7cf1b141bc8586b94cf6f34aba]
I20250623 08:04:29.377096 710 raft_consensus.cc:2802] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.377239 710 raft_consensus.cc:695] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:29.377514 710 consensus_queue.cc:237] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.377869 731 catalog_manager.cc:671] Loaded metadata for table default.Table_fd49e04fcac14f0194cc9fed910601fc [id=17b055700da64cc1b19f93dd5edf40d5]
I20250623 08:04:29.378304 731 catalog_manager.cc:671] Loaded metadata for table default.Table_9fc75fd22c6a467782fe3b4c3b123602 [id=1a49c0c47b6647ef8fc4c7c20ad9bc1b]
I20250623 08:04:29.379130 731 catalog_manager.cc:671] Loaded metadata for table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]
I20250623 08:04:29.379619 731 catalog_manager.cc:671] Loaded metadata for table default.Table_a615b39019e1478aa69bd46fc2e4a4a9 [id=1e194b3160b64acabac38acfd50a64b3]
I20250623 08:04:29.380352 731 catalog_manager.cc:671] Loaded metadata for table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]
I20250623 08:04:29.381156 731 catalog_manager.cc:671] Loaded metadata for table default.Table_9f3d2a1f5b3c41fc888725e2b7d45e0e [id=25eb4e3f3a6e42e988d2a3613ef4575f]
I20250623 08:04:29.386121 731 catalog_manager.cc:671] Loaded metadata for table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]
I20250623 08:04:29.386197 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "175ee90133994154b7e4b58b0770db3b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.386402 32379 raft_consensus.cc:3058] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.387648 731 catalog_manager.cc:671] Loaded metadata for table default.Table_4a19c65595654fa98efc42626b6a9d85 [id=2979e4296458439ea5753b6ba67634ac]
I20250623 08:04:29.387809 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "175ee90133994154b7e4b58b0770db3b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.388072 32653 raft_consensus.cc:3058] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.388280 731 catalog_manager.cc:671] Loaded metadata for table default.Table_af3136dbadfb43c892a197cc3a5d684e [id=2bfacaaab9f347b1a77e5eb29a9c485e]
I20250623 08:04:29.389046 731 catalog_manager.cc:671] Loaded metadata for table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]
I20250623 08:04:29.389580 731 catalog_manager.cc:671] Loaded metadata for table default.Table_a6ef1bf66cd24835896dabd3af080a6a [id=33768fa88bfc4cfba1359030af5e77c9]
I20250623 08:04:29.389642 32379 raft_consensus.cc:2466] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:29.390676 731 catalog_manager.cc:671] Loaded metadata for table default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]
I20250623 08:04:29.391192 731 catalog_manager.cc:671] Loaded metadata for table default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c [id=387b3fc3d26b457099bcbb3b8ef3ac9b]
I20250623 08:04:29.391213 342 tablet_bootstrap.cc:492] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.391367 342 ts_tablet_manager.cc:1397] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.105s user 0.017s sys 0.008s
I20250623 08:04:29.391758 731 catalog_manager.cc:671] Loaded metadata for table default.Table_e14bd6f5e20e42b1898bce238d4d986c [id=3a0c95fd4bd84b80ad1f5951cbef92e8]
I20250623 08:04:29.391880 32653 raft_consensus.cc:2466] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:29.392305 731 catalog_manager.cc:671] Loaded metadata for table default.Table_fae6ac5a256346149e8cf56ae9dbdf3b [id=3c89160b4fb54f649fd3d222f6a6699d]
I20250623 08:04:29.392292 342 raft_consensus.cc:357] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.392488 342 raft_consensus.cc:383] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.392599 342 raft_consensus.cc:738] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.392886 342 consensus_queue.cc:260] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.393450 342 ts_tablet_manager.cc:1428] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:29.393801 342 tablet_bootstrap.cc:492] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.394420 32453 leader_election.cc:304] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.395721 342 tablet_bootstrap.cc:654] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.396179 657 raft_consensus.cc:2802] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.396346 657 raft_consensus.cc:695] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:29.396616 657 consensus_queue.cc:237] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.400060 339 tablet_bootstrap.cc:492] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.400243 339 ts_tablet_manager.cc:1397] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.113s user 0.022s sys 0.029s
I20250623 08:04:29.401077 339 raft_consensus.cc:357] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.401291 339 raft_consensus.cc:383] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.401393 339 raft_consensus.cc:738] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.401667 339 consensus_queue.cc:260] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.403504 710 raft_consensus.cc:491] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.403776 710 raft_consensus.cc:513] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.404531 710 leader_election.cc:290] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.405285 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ee5a4fe03848229524cc0105226d8c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.405625 32379 raft_consensus.cc:2391] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:29.406055 339 ts_tablet_manager.cc:1428] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:29.406533 339 tablet_bootstrap.cc:492] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.408061 710 raft_consensus.cc:491] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.408221 710 raft_consensus.cc:513] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.408797 710 leader_election.cc:290] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.409085 731 catalog_manager.cc:671] Loaded metadata for table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]
I20250623 08:04:29.409317 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "effd4fff43fe4d4fb36458986e1dfce1" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:29.411860 32589 leader_election.cc:343] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: d0ee5a4fe03848229524cc0105226d8c
I20250623 08:04:29.412756 32589 leader_election.cc:304] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:29.413357 731 catalog_manager.cc:671] Loaded metadata for table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]
I20250623 08:04:29.414274 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "effd4fff43fe4d4fb36458986e1dfce1" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.414559 32518 raft_consensus.cc:2391] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:29.414532 710 raft_consensus.cc:3058] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.415393 32589 leader_election.cc:304] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:29.415621 339 tablet_bootstrap.cc:654] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.415841 819 raft_consensus.cc:3058] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.417862 343 tablet_bootstrap.cc:492] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.418013 343 ts_tablet_manager.cc:1397] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.014s
I20250623 08:04:29.418768 819 raft_consensus.cc:2747] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:29.418902 343 raft_consensus.cc:357] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.419085 343 raft_consensus.cc:383] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.419221 343 raft_consensus.cc:738] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.419467 343 consensus_queue.cc:260] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.420025 343 ts_tablet_manager.cc:1428] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:29.420317 343 tablet_bootstrap.cc:492] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.418785 731 catalog_manager.cc:671] Loaded metadata for table default.Table_c382db2a0c144ec4abd51b3e927e0a17 [id=461d38c05dc24245b0c59cb73b42a394]
I20250623 08:04:29.422125 343 tablet_bootstrap.cc:654] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.424090 731 catalog_manager.cc:671] Loaded metadata for table default.Table_4e377b569121400d82096403060f9cda [id=49bfa95de97b443ba51225f84820fcf2]
I20250623 08:04:29.424255 710 raft_consensus.cc:2747] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:29.425222 731 catalog_manager.cc:671] Loaded metadata for table default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]
I20250623 08:04:29.428653 731 catalog_manager.cc:671] Loaded metadata for table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]
I20250623 08:04:29.430140 731 catalog_manager.cc:671] Loaded metadata for table default.Table_de3536331f614b4a8de796d3328c9fa2 [id=5545b396fc5746469e3a8908b0178a4c]
I20250623 08:04:29.434047 731 catalog_manager.cc:671] Loaded metadata for table default.Table_4550e74cc6e845719c1466d3acd1b723 [id=587d6896c7dd40838ce8ad01026379a8]
I20250623 08:04:29.439499 699 consensus_queue.cc:1035] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 08:04:29.445843 710 raft_consensus.cc:491] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.446053 710 raft_consensus.cc:513] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.447098 710 leader_election.cc:290] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.447968 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485538f7c63948818f20a25c02381519" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.448232 32518 raft_consensus.cc:2391] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:29.448330 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485538f7c63948818f20a25c02381519" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.449322 32589 leader_election.cc:304] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:29.450184 710 raft_consensus.cc:3058] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.453114 710 raft_consensus.cc:2747] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:29.459626 342 tablet_bootstrap.cc:492] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.459806 342 ts_tablet_manager.cc:1397] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.015s
I20250623 08:04:29.460577 342 raft_consensus.cc:357] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.460813 342 raft_consensus.cc:383] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.460906 342 raft_consensus.cc:738] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.461126 342 consensus_queue.cc:260] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.461673 342 ts_tablet_manager.cc:1428] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:29.461756 343 tablet_bootstrap.cc:492] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.461901 343 ts_tablet_manager.cc:1397] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.008s
I20250623 08:04:29.461987 342 tablet_bootstrap.cc:492] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.463562 343 raft_consensus.cc:357] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.463766 339 tablet_bootstrap.cc:492] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.463757 343 raft_consensus.cc:383] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.463840 343 raft_consensus.cc:738] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.463933 339 ts_tablet_manager.cc:1397] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.012s
I20250623 08:04:29.464094 343 consensus_queue.cc:260] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.464566 343 ts_tablet_manager.cc:1428] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:29.464682 339 raft_consensus.cc:357] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.464881 339 raft_consensus.cc:383] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.464968 339 raft_consensus.cc:738] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.465170 339 consensus_queue.cc:260] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.465951 339 ts_tablet_manager.cc:1428] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:29.466221 339 tablet_bootstrap.cc:492] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.466676 343 tablet_bootstrap.cc:492] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.467962 339 tablet_bootstrap.cc:654] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.468555 343 tablet_bootstrap.cc:654] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.480473 819 consensus_queue.cc:1035] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.480742 731 catalog_manager.cc:671] Loaded metadata for table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]
I20250623 08:04:29.481300 731 catalog_manager.cc:671] Loaded metadata for table default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]
I20250623 08:04:29.481962 731 catalog_manager.cc:671] Loaded metadata for table default.Table_fe75261182e243ebbbcfbf6e636ba134 [id=5d4bcecfd3904412b0aa6fab7c75d584]
I20250623 08:04:29.483399 342 tablet_bootstrap.cc:654] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.485283 342 tablet_bootstrap.cc:492] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.485410 342 ts_tablet_manager.cc:1397] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250623 08:04:29.486131 342 raft_consensus.cc:357] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.486307 342 raft_consensus.cc:383] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.486392 342 raft_consensus.cc:738] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.486618 342 consensus_queue.cc:260] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.487082 342 ts_tablet_manager.cc:1428] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:29.488003 342 tablet_bootstrap.cc:492] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.489727 342 tablet_bootstrap.cc:654] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.493659 339 tablet_bootstrap.cc:492] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.493829 339 ts_tablet_manager.cc:1397] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250623 08:04:29.494591 339 raft_consensus.cc:357] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.494793 339 raft_consensus.cc:383] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.494876 339 raft_consensus.cc:738] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.495105 339 consensus_queue.cc:260] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.495581 339 ts_tablet_manager.cc:1428] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:29.495920 339 tablet_bootstrap.cc:492] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.498145 342 tablet_bootstrap.cc:492] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.498299 342 ts_tablet_manager.cc:1397] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250623 08:04:29.499138 342 raft_consensus.cc:357] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.499303 339 tablet_bootstrap.cc:654] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.499351 342 raft_consensus.cc:383] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.499439 342 raft_consensus.cc:738] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.499451 343 tablet_bootstrap.cc:492] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.499593 343 ts_tablet_manager.cc:1397] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20250623 08:04:29.499701 342 consensus_queue.cc:260] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.500183 342 ts_tablet_manager.cc:1428] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:29.500505 342 tablet_bootstrap.cc:492] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.500392 343 raft_consensus.cc:357] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.500581 343 raft_consensus.cc:383] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.500675 343 raft_consensus.cc:738] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.500918 343 consensus_queue.cc:260] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.503347 342 tablet_bootstrap.cc:654] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.504985 343 ts_tablet_manager.cc:1428] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250623 08:04:29.505632 343 tablet_bootstrap.cc:492] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.508183 343 tablet_bootstrap.cc:654] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.510157 848 consensus_queue.cc:1035] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.511142 848 raft_consensus.cc:491] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.511368 848 raft_consensus.cc:513] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.511618 342 tablet_bootstrap.cc:492] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.511760 342 ts_tablet_manager.cc:1397] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20250623 08:04:29.512208 848 leader_election.cc:290] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.512526 342 raft_consensus.cc:357] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.512702 342 raft_consensus.cc:383] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.512785 342 raft_consensus.cc:738] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.512977 342 consensus_queue.cc:260] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.513451 342 ts_tablet_manager.cc:1428] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:29.513768 342 tablet_bootstrap.cc:492] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.516961 342 tablet_bootstrap.cc:654] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.518309 610 raft_consensus.cc:491] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.518462 610 raft_consensus.cc:513] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.519150 610 leader_election.cc:290] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.519979 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2795cb14e6db4116a4c9ee539bd4e82d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.519995 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2795cb14e6db4116a4c9ee539bd4e82d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.520203 32518 raft_consensus.cc:2466] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:29.520263 32654 raft_consensus.cc:2466] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:29.520048 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cde7df7d39a54114a4dcb28cfd1eb429" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.520565 32653 raft_consensus.cc:2466] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:29.521695 32314 leader_election.cc:304] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.522248 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cde7df7d39a54114a4dcb28cfd1eb429" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.522445 610 raft_consensus.cc:2802] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.522480 32379 raft_consensus.cc:2466] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:29.522567 610 raft_consensus.cc:491] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.522670 343 tablet_bootstrap.cc:492] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.522698 610 raft_consensus.cc:3058] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.522861 343 ts_tablet_manager.cc:1397] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20250623 08:04:29.523844 343 raft_consensus.cc:357] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.524072 343 raft_consensus.cc:383] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.524152 343 raft_consensus.cc:738] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.524471 343 consensus_queue.cc:260] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.525120 32453 leader_election.cc:304] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.525600 610 raft_consensus.cc:513] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.525825 874 raft_consensus.cc:2802] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.525877 343 ts_tablet_manager.cc:1428] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:29.525921 874 raft_consensus.cc:491] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.526029 874 raft_consensus.cc:3058] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.526216 342 tablet_bootstrap.cc:492] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.526331 343 tablet_bootstrap.cc:492] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.526367 342 ts_tablet_manager.cc:1397] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20250623 08:04:29.527175 342 raft_consensus.cc:357] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.527213 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2795cb14e6db4116a4c9ee539bd4e82d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.527326 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2795cb14e6db4116a4c9ee539bd4e82d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.527393 342 raft_consensus.cc:383] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.527413 32653 raft_consensus.cc:3058] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.527496 342 raft_consensus.cc:738] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.527503 32518 raft_consensus.cc:3058] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.527832 342 consensus_queue.cc:260] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.528358 342 ts_tablet_manager.cc:1428] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:29.528687 342 tablet_bootstrap.cc:492] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.530350 32653 raft_consensus.cc:2466] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.530609 342 tablet_bootstrap.cc:654] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.532402 343 tablet_bootstrap.cc:654] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.531852 32314 leader_election.cc:304] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.533165 685 raft_consensus.cc:2802] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.533300 685 raft_consensus.cc:695] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:29.533558 685 consensus_queue.cc:237] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.533794 342 tablet_bootstrap.cc:492] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.533994 342 ts_tablet_manager.cc:1397] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250623 08:04:29.534905 342 raft_consensus.cc:357] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.535082 342 raft_consensus.cc:383] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.535168 342 raft_consensus.cc:738] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.535387 342 consensus_queue.cc:260] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.530699 32518 raft_consensus.cc:2466] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.535988 342 ts_tablet_manager.cc:1428] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:29.536271 342 tablet_bootstrap.cc:492] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.536885 746 raft_consensus.cc:491] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.537027 746 raft_consensus.cc:513] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.537634 746 leader_election.cc:290] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.537894 342 tablet_bootstrap.cc:654] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.538794 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba3d2fa01d574dd992dc904ac024ba3a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:29.539721 32314 leader_election.cc:343] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.540187 342 tablet_bootstrap.cc:492] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.540347 342 ts_tablet_manager.cc:1397] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250623 08:04:29.540489 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba3d2fa01d574dd992dc904ac024ba3a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.541088 32518 raft_consensus.cc:2466] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:29.541133 342 raft_consensus.cc:357] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.541630 342 raft_consensus.cc:383] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.541721 342 raft_consensus.cc:738] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.541860 32314 leader_election.cc:304] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:29.541944 342 consensus_queue.cc:260] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.542228 746 raft_consensus.cc:2802] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.542363 746 raft_consensus.cc:491] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.542447 746 raft_consensus.cc:3058] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.542454 342 ts_tablet_manager.cc:1428] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:29.542804 848 raft_consensus.cc:491] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.542923 848 raft_consensus.cc:513] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.543501 343 tablet_bootstrap.cc:492] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.543633 343 ts_tablet_manager.cc:1397] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250623 08:04:29.543677 848 leader_election.cc:290] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.544267 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d02f74c24a174b18b08a870a4e47fefb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.544520 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d02f74c24a174b18b08a870a4e47fefb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.544694 731 catalog_manager.cc:671] Loaded metadata for table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]
I20250623 08:04:29.544970 343 raft_consensus.cc:357] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:29.545097 32453 leader_election.cc:343] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.545150 343 raft_consensus.cc:383] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.545266 343 raft_consensus.cc:738] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:29.545296 32453 leader_election.cc:343] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.545382 32453 leader_election.cc:304] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:29.545508 343 consensus_queue.cc:260] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.545722 848 raft_consensus.cc:2747] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:29.546109 343 ts_tablet_manager.cc:1428] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:29.546393 343 tablet_bootstrap.cc:492] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.547870 746 raft_consensus.cc:513] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.548430 746 leader_election.cc:290] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.548518 342 tablet_bootstrap.cc:492] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.548637 32020 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskJxinPB/build/asan/bin/kudu
/tmp/dist-test-taskJxinPB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.61:33313
--webserver_interface=127.31.69.61
--webserver_port=44043
--builtin_ntp_servers=127.31.69.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250623 08:04:29.549403 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba3d2fa01d574dd992dc904ac024ba3a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
W20250623 08:04:29.549981 32314 leader_election.cc:343] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.550355 731 catalog_manager.cc:671] Loaded metadata for table default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac [id=65c4e79e095b41b2b2a42e9858485423]
I20250623 08:04:29.550429 342 tablet_bootstrap.cc:654] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.526237 610 leader_election.cc:290] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.550951 731 catalog_manager.cc:671] Loaded metadata for table default.Table_e2502c251f3c4a7f8caa4faab352ebda [id=664b85653cd341a6a6afe7d04fb9c2ac]
I20250623 08:04:29.551492 731 catalog_manager.cc:671] Loaded metadata for table default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]
I20250623 08:04:29.552019 343 tablet_bootstrap.cc:654] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.552019 731 catalog_manager.cc:671] Loaded metadata for table default.Table_4acf3aff222e477a885bc37d00623ddf [id=6c57db68f5ed45a3b64d1e3a02ce8a2a]
I20250623 08:04:29.555794 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba3d2fa01d574dd992dc904ac024ba3a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.556707 32518 raft_consensus.cc:3058] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.558673 610 raft_consensus.cc:491] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.559281 610 raft_consensus.cc:513] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.560021 610 leader_election.cc:290] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.560524 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417a050eada4403ebbddff255fb6fb5a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.560709 32517 raft_consensus.cc:2466] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:29.561100 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417a050eada4403ebbddff255fb6fb5a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.561496 32314 leader_election.cc:304] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.553782 874 raft_consensus.cc:513] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.562126 610 raft_consensus.cc:2802] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.562335 610 raft_consensus.cc:491] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 08:04:29.562819 32314 leader_election.cc:343] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.563412 610 raft_consensus.cc:3058] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.564256 731 catalog_manager.cc:671] Loaded metadata for table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]
I20250623 08:04:29.564827 731 catalog_manager.cc:671] Loaded metadata for table default.Table_6e40e423312e4f9e95174f277b8897ce [id=7038d3807ece414f8a8e76bfe107c3b3]
I20250623 08:04:29.565323 731 catalog_manager.cc:671] Loaded metadata for table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]
I20250623 08:04:29.565851 731 catalog_manager.cc:671] Loaded metadata for table default.Table_c2afde03c007450390a0466fe1d39344 [id=77e1411772b54c89a9a8435654249287]
I20250623 08:04:29.566354 731 catalog_manager.cc:671] Loaded metadata for table default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac [id=793edd0cb585499197d2f0e57a83be84]
I20250623 08:04:29.568804 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cde7df7d39a54114a4dcb28cfd1eb429" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.569012 32656 raft_consensus.cc:3058] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.562686 874 leader_election.cc:290] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.569854 731 catalog_manager.cc:671] Loaded metadata for table default.Table_463abc7f8321496da24ccb06cba92d25 [id=7a569a9fe3bd4c608ddca48a734df1da]
I20250623 08:04:29.570077 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cde7df7d39a54114a4dcb28cfd1eb429" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.570242 32379 raft_consensus.cc:3058] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.570533 731 catalog_manager.cc:671] Loaded metadata for table default.Table_1da3c8ca68084386b7f0d95a6f85927a [id=7cc0919082d04fc4a02aa856a42e5529]
I20250623 08:04:29.572059 731 catalog_manager.cc:671] Loaded metadata for table default.Table_bc69503004f74bf59fd961b4f209baa6 [id=836b8d720fae4622a556600a1a9c1d91]
I20250623 08:04:29.572131 32656 raft_consensus.cc:2466] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:29.572470 731 catalog_manager.cc:671] Loaded metadata for table default.Table_0bb2584ec3e0467194a9591bbe5ae477 [id=87e7893d85264d4c9cbcc5564f840390]
I20250623 08:04:29.572822 32518 raft_consensus.cc:2466] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.573225 32314 leader_election.cc:304] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:29.573899 746 raft_consensus.cc:2802] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.574108 32379 raft_consensus.cc:2466] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:29.573994 610 raft_consensus.cc:513] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.574709 731 catalog_manager.cc:671] Loaded metadata for table default.Table_fe81102858f244af9a4a286eaf6e34bf [id=8a0d9f5619d843148169561ffc7cfdc0]
I20250623 08:04:29.575198 610 leader_election.cc:290] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.575373 731 catalog_manager.cc:671] Loaded metadata for table default.Table_7e39ecbac4a8433e8ae98b0db22c30ea [id=8b20894a91c54a3eb08d5e1b4a42d269]
I20250623 08:04:29.576251 731 catalog_manager.cc:671] Loaded metadata for table default.Table_61ccd9db16cb4618874bff73413fd485 [id=8d6f6785bcf24b719419015368a05f68]
I20250623 08:04:29.576658 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417a050eada4403ebbddff255fb6fb5a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.576866 32518 raft_consensus.cc:3058] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.576911 731 catalog_manager.cc:671] Loaded metadata for table default.Table_f1d8f7ef7cf144eaa1fd741b10eac3d6 [id=9060c1640fca461c9020722b3165d144]
I20250623 08:04:29.577739 731 catalog_manager.cc:671] Loaded metadata for table default.Table_e69f9fec5b604bca8658b2124145f92e [id=96cf904f9ea64544ae6f662208874f89]
I20250623 08:04:29.578275 731 catalog_manager.cc:671] Loaded metadata for table default.Table_7987e1a82b644bd69724bc61cd84445a [id=96f6aedcc9704226b4b14e5548a35ad1]
I20250623 08:04:29.579319 731 catalog_manager.cc:671] Loaded metadata for table default.Table_e962b9945a5041039f7e33b98b653175 [id=9d9e232100264e378516c06c61310750]
I20250623 08:04:29.580402 731 catalog_manager.cc:671] Loaded metadata for table default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 [id=9f97f6deca194137a353ea221d0e3676]
I20250623 08:04:29.580469 32518 raft_consensus.cc:2466] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.581002 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417a050eada4403ebbddff255fb6fb5a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.581698 731 catalog_manager.cc:671] Loaded metadata for table default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]
W20250623 08:04:29.581662 32314 leader_election.cc:343] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.581912 32314 leader_election.cc:304] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:29.582219 610 raft_consensus.cc:2802] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.582486 610 raft_consensus.cc:695] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:29.582864 610 consensus_queue.cc:237] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.583241 731 catalog_manager.cc:671] Loaded metadata for table default.Table_c4287575a32b4847ba2379ef60074ba9 [id=abda89a30971466ea9a1f410d424ef58]
I20250623 08:04:29.584697 731 catalog_manager.cc:671] Loaded metadata for table default.Table_8ef30265c1894796843272f538aaa79e [id=ad13fb2c08c448bebe3e65b0143d66bc]
I20250623 08:04:29.586445 731 catalog_manager.cc:671] Loaded metadata for table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]
I20250623 08:04:29.587077 731 catalog_manager.cc:671] Loaded metadata for table default.Table_244bcdc8d69b4f61a337dac403ffe6cb [id=b3ab55b66d754642aed2f746d9a35807]
I20250623 08:04:29.587894 731 catalog_manager.cc:671] Loaded metadata for table default.Table_d3f72251faee462c9c301b8aff464bb5 [id=b7671dfa27be46bb8b811b12016434e2]
I20250623 08:04:29.588660 731 catalog_manager.cc:671] Loaded metadata for table default.Table_2b1942032adc43d78d1f9e83417b42eb [id=b827ea08f0b14c9aa2b2c3d02d15f150]
I20250623 08:04:29.589246 731 catalog_manager.cc:671] Loaded metadata for table default.Table_1671bbefe7bd4d4b86a917373288a1a6 [id=baaeae53fdcb49b7889d8379fa203c79]
I20250623 08:04:29.592043 339 tablet_bootstrap.cc:492] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.592140 342 tablet_bootstrap.cc:492] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.592237 339 ts_tablet_manager.cc:1397] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.000s
I20250623 08:04:29.592420 342 ts_tablet_manager.cc:1397] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.011s
I20250623 08:04:29.593196 339 raft_consensus.cc:357] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.593199 342 raft_consensus.cc:357] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.593387 339 raft_consensus.cc:383] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.593418 342 raft_consensus.cc:383] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.593463 339 raft_consensus.cc:738] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.593498 342 raft_consensus.cc:738] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.593713 339 consensus_queue.cc:260] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.593784 32453 leader_election.cc:304] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:29.593894 342 consensus_queue.cc:260] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.594054 657 raft_consensus.cc:2802] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.594282 657 raft_consensus.cc:695] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:29.594772 342 ts_tablet_manager.cc:1428] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:29.594728 657 consensus_queue.cc:237] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.595255 342 tablet_bootstrap.cc:492] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.595865 731 catalog_manager.cc:671] Loaded metadata for table default.Table_75451c4863a34537a328617c53d46cae [id=be88c6ecf20d4ab1b3757c27ce998f87]
I20250623 08:04:29.596415 731 catalog_manager.cc:671] Loaded metadata for table default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]
I20250623 08:04:29.596923 731 catalog_manager.cc:671] Loaded metadata for table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]
I20250623 08:04:29.597651 342 tablet_bootstrap.cc:654] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.599668 343 tablet_bootstrap.cc:492] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.599833 343 ts_tablet_manager.cc:1397] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.013s
I20250623 08:04:29.600201 731 catalog_manager.cc:671] Loaded metadata for table default.Table_79be3d7247284828a82f02d96faa60cc [id=c618740b8cb040bcb78c452b3d747391]
I20250623 08:04:29.600723 731 catalog_manager.cc:671] Loaded metadata for table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]
I20250623 08:04:29.600670 343 raft_consensus.cc:357] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.600915 343 raft_consensus.cc:383] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.600996 343 raft_consensus.cc:738] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.601274 343 consensus_queue.cc:260] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.601827 343 ts_tablet_manager.cc:1428] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:29.602068 731 catalog_manager.cc:671] Loaded metadata for table default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517]
I20250623 08:04:29.602162 343 tablet_bootstrap.cc:492] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.602878 731 catalog_manager.cc:671] Loaded metadata for table default.Table_e851e10456a04a51814a556167f4777f [id=ced0b4cf62924dbfadad02109349bea0]
I20250623 08:04:29.603250 746 raft_consensus.cc:695] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:29.603566 731 catalog_manager.cc:671] Loaded metadata for table default.Table_79f69fca0b25496e896d93fdd78e14b6 [id=cf095cc479d741b8b7c2cd352724713a]
I20250623 08:04:29.603626 746 consensus_queue.cc:237] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.603788 893 consensus_queue.cc:1035] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.604044 343 tablet_bootstrap.cc:654] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.604528 731 catalog_manager.cc:671] Loaded metadata for table default.Table_6b0157583afb475b863719603a5ba38d [id=d1fbcd8e89164d71b021f4b1f4223a59]
I20250623 08:04:29.605065 731 catalog_manager.cc:671] Loaded metadata for table default.Table_640be163e2114433bd7079aa3a602116 [id=d422f6202d1842e998a23ec064757334]
I20250623 08:04:29.606103 731 catalog_manager.cc:671] Loaded metadata for table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]
I20250623 08:04:29.607451 731 catalog_manager.cc:671] Loaded metadata for table default.Table_3ef2e86e99164cce8e882e6954e2ec17 [id=dbbd669a72954d1aaf169b5870a85fc3]
I20250623 08:04:29.608543 731 catalog_manager.cc:671] Loaded metadata for table default.Table_d226f9dab24f45ae896579983b3d1a67 [id=e0cd5a7f63b443c2a3e86ec75a1e3778]
I20250623 08:04:29.609593 731 catalog_manager.cc:671] Loaded metadata for table default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]
I20250623 08:04:29.609813 342 tablet_bootstrap.cc:492] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.610004 342 ts_tablet_manager.cc:1397] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20250623 08:04:29.610788 342 raft_consensus.cc:357] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.610989 342 raft_consensus.cc:383] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.611063 342 raft_consensus.cc:738] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.611275 342 consensus_queue.cc:260] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.611914 342 ts_tablet_manager.cc:1428] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:29.612246 342 tablet_bootstrap.cc:492] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.612387 731 catalog_manager.cc:671] Loaded metadata for table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]
I20250623 08:04:29.614037 342 tablet_bootstrap.cc:654] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.616534 731 catalog_manager.cc:671] Loaded metadata for table default.Table_2c99afea2b6e4963985dab32b545779e [id=e6cd13b44efa4cbbbdd2c14d8ea22267]
I20250623 08:04:29.617630 731 catalog_manager.cc:671] Loaded metadata for table default.Table_5f159a56c60144b99a6847ec6080623e [id=ec1f9e35e20c47fcbe2d4b80c7abeabc]
I20250623 08:04:29.617969 339 ts_tablet_manager.cc:1428] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250623 08:04:29.618305 339 tablet_bootstrap.cc:492] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.619014 731 catalog_manager.cc:671] Loaded metadata for table default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c]
I20250623 08:04:29.619150 343 tablet_bootstrap.cc:492] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.619300 343 ts_tablet_manager.cc:1397] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20250623 08:04:29.619637 731 catalog_manager.cc:671] Loaded metadata for table default.Table_48cbdc58fd034e6390087ed7c6346d24 [id=ef1a8122b8044388b4ae24a03da26837]
I20250623 08:04:29.620213 343 raft_consensus.cc:357] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.620404 343 raft_consensus.cc:383] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.620527 343 raft_consensus.cc:738] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.620782 343 consensus_queue.cc:260] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.621393 343 ts_tablet_manager.cc:1428] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:29.621690 343 tablet_bootstrap.cc:492] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.622869 731 catalog_manager.cc:671] Loaded metadata for table default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b]
I20250623 08:04:29.623167 342 tablet_bootstrap.cc:492] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.623319 342 ts_tablet_manager.cc:1397] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250623 08:04:29.622629 896 consensus_queue.cc:1035] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 08:04:29.624899 343 tablet_bootstrap.cc:654] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.626842 699 consensus_queue.cc:1035] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.627177 342 raft_consensus.cc:357] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.627357 342 raft_consensus.cc:383] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.627457 342 raft_consensus.cc:738] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.627588 339 tablet_bootstrap.cc:654] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.627696 342 consensus_queue.cc:260] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.628228 342 ts_tablet_manager.cc:1428] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250623 08:04:29.628530 342 tablet_bootstrap.cc:492] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.630187 342 tablet_bootstrap.cc:654] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.633572 731 tablet_loader.cc:96] loaded metadata for tablet 025365d2fa0a43dd983d395a05cc2af1 (table default.Table_2c99afea2b6e4963985dab32b545779e [id=e6cd13b44efa4cbbbdd2c14d8ea22267])
I20250623 08:04:29.634202 731 tablet_loader.cc:96] loaded metadata for tablet 0311ce67989f49e8866234a88e5b4d09 (table default.Table_e962b9945a5041039f7e33b98b653175 [id=9d9e232100264e378516c06c61310750])
I20250623 08:04:29.635525 731 tablet_loader.cc:96] loaded metadata for tablet 0545daa6fba846adbc573a63d8dbcd66 (table default.Table_e69f9fec5b604bca8658b2124145f92e [id=96cf904f9ea64544ae6f662208874f89])
I20250623 08:04:29.636215 731 tablet_loader.cc:96] loaded metadata for tablet 0861107ab4624a05bd7a4d24b420c502 (table default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb])
I20250623 08:04:29.636627 731 tablet_loader.cc:96] loaded metadata for tablet 0861e45cf546487d9812caa06e2704eb (table default.Table_1671bbefe7bd4d4b86a917373288a1a6 [id=baaeae53fdcb49b7889d8379fa203c79])
I20250623 08:04:29.637236 731 tablet_loader.cc:96] loaded metadata for tablet 0c58231f67604422a1497e4c1b44ec52 (table default.Table_fe81102858f244af9a4a286eaf6e34bf [id=8a0d9f5619d843148169561ffc7cfdc0])
I20250623 08:04:29.637892 342 tablet_bootstrap.cc:492] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.638038 342 ts_tablet_manager.cc:1397] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20250623 08:04:29.638454 731 tablet_loader.cc:96] loaded metadata for tablet 0db9480bb07343fe935fe9bfc98bca77 (table default.Table_6b0157583afb475b863719603a5ba38d [id=d1fbcd8e89164d71b021f4b1f4223a59])
I20250623 08:04:29.638854 731 tablet_loader.cc:96] loaded metadata for tablet 0e9bf7cba2244703b7402cb4be1a005e (table default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c])
I20250623 08:04:29.638852 342 raft_consensus.cc:357] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.639060 342 raft_consensus.cc:383] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.639142 342 raft_consensus.cc:738] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.639354 342 consensus_queue.cc:260] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.641237 342 ts_tablet_manager.cc:1428] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:29.641530 342 tablet_bootstrap.cc:492] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.641824 610 consensus_queue.cc:1035] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.642838 731 tablet_loader.cc:96] loaded metadata for tablet 136e482995a24c5daec55a701f3091fa (table default.Table_fae6ac5a256346149e8cf56ae9dbdf3b [id=3c89160b4fb54f649fd3d222f6a6699d])
I20250623 08:04:29.643523 342 tablet_bootstrap.cc:654] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.646605 343 tablet_bootstrap.cc:492] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.646821 343 ts_tablet_manager.cc:1397] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20250623 08:04:29.647678 819 consensus_queue.cc:1035] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.652091 731 tablet_loader.cc:96] loaded metadata for tablet 1a070895f4984493a3b9f6fa779797a9 (table default.Table_79f69fca0b25496e896d93fdd78e14b6 [id=cf095cc479d741b8b7c2cd352724713a])
I20250623 08:04:29.652700 343 raft_consensus.cc:357] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.652931 343 raft_consensus.cc:383] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.653016 343 raft_consensus.cc:738] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.653236 343 consensus_queue.cc:260] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.654686 343 ts_tablet_manager.cc:1428] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250623 08:04:29.655505 343 tablet_bootstrap.cc:492] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.657238 343 tablet_bootstrap.cc:654] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.657464 731 tablet_loader.cc:96] loaded metadata for tablet 20e7a78cfc384033810603316b9e66bf (table default.Table_7987e1a82b644bd69724bc61cd84445a [id=96f6aedcc9704226b4b14e5548a35ad1])
I20250623 08:04:29.657904 731 tablet_loader.cc:96] loaded metadata for tablet 21080ab069d74457a9f7fe3aabdb3c7c (table default.Table_f1d8f7ef7cf144eaa1fd741b10eac3d6 [id=9060c1640fca461c9020722b3165d144])
I20250623 08:04:29.658782 731 tablet_loader.cc:96] loaded metadata for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (table default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b])
I20250623 08:04:29.659368 731 tablet_loader.cc:96] loaded metadata for tablet 26f4a1b472e848ccb9a1ec71c1dd3516 (table default.Table_61ccd9db16cb4618874bff73413fd485 [id=8d6f6785bcf24b719419015368a05f68])
I20250623 08:04:29.660605 731 tablet_loader.cc:96] loaded metadata for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec])
I20250623 08:04:29.661653 731 tablet_loader.cc:96] loaded metadata for tablet 2f62ec83c88e43438b963e740d2c7004 (table default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac [id=65c4e79e095b41b2b2a42e9858485423])
I20250623 08:04:29.662113 342 tablet_bootstrap.cc:492] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.662111 731 tablet_loader.cc:96] loaded metadata for tablet 31a67b70cbc64bb3bfd5a34865109a01 (table default.Table_463abc7f8321496da24ccb06cba92d25 [id=7a569a9fe3bd4c608ddca48a734df1da])
I20250623 08:04:29.662267 342 ts_tablet_manager.cc:1397] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20250623 08:04:29.662829 731 tablet_loader.cc:96] loaded metadata for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca])
I20250623 08:04:29.663215 342 raft_consensus.cc:357] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.663412 342 raft_consensus.cc:383] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.663460 893 raft_consensus.cc:491] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.663516 342 raft_consensus.cc:738] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.663661 893 raft_consensus.cc:513] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.663991 731 tablet_loader.cc:96] loaded metadata for tablet 3249605158634f4eab5b9d61357c383f (table default.Table_79be3d7247284828a82f02d96faa60cc [id=c618740b8cb040bcb78c452b3d747391])
I20250623 08:04:29.665072 893 leader_election.cc:290] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.665588 731 tablet_loader.cc:96] loaded metadata for tablet 358cb6ac24414bdfbc32155e732f9ac8 (table default.Table_2b1942032adc43d78d1f9e83417b42eb [id=b827ea08f0b14c9aa2b2c3d02d15f150])
I20250623 08:04:29.666024 343 tablet_bootstrap.cc:492] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.666188 343 ts_tablet_manager.cc:1397] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20250623 08:04:29.666564 731 tablet_loader.cc:96] loaded metadata for tablet 38ecd4c243b840f4b66d8e01078dca64 (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9])
I20250623 08:04:29.667127 343 raft_consensus.cc:357] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.667351 343 raft_consensus.cc:383] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.667444 343 raft_consensus.cc:738] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.667699 343 consensus_queue.cc:260] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.668208 731 tablet_loader.cc:96] loaded metadata for tablet 432d1642c0e44119b5bc05858463a136 (table default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297])
I20250623 08:04:29.668677 731 tablet_loader.cc:96] loaded metadata for tablet 485538f7c63948818f20a25c02381519 (table default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873])
I20250623 08:04:29.669118 731 tablet_loader.cc:96] loaded metadata for tablet 48da1e4759bb4fea87a45ad8a428b6e7 (table default.Table_c382db2a0c144ec4abd51b3e927e0a17 [id=461d38c05dc24245b0c59cb73b42a394])
I20250623 08:04:29.669610 731 tablet_loader.cc:96] loaded metadata for tablet 4ac9f38877b24d619c280a1aaa0b7b08 (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee])
I20250623 08:04:29.669759 342 consensus_queue.cc:260] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.670373 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4221d7500d1458db75926d7b274a654" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.670518 342 ts_tablet_manager.cc:1428] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250623 08:04:29.670559 32654 raft_consensus.cc:2466] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:29.670894 342 tablet_bootstrap.cc:492] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.670979 731 tablet_loader.cc:96] loaded metadata for tablet 4fe30079045d437aa61bb5bfbfb96133 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20])
I20250623 08:04:29.671286 731 tablet_loader.cc:96] loaded metadata for tablet 52d0a356c74f4daea9093c998b4bf73e (table default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec])
I20250623 08:04:29.671700 32314 leader_election.cc:304] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.672477 610 raft_consensus.cc:2802] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.672955 610 raft_consensus.cc:491] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.673069 343 ts_tablet_manager.cc:1428] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:29.673077 610 raft_consensus.cc:3058] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.673379 343 tablet_bootstrap.cc:492] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.675206 343 tablet_bootstrap.cc:654] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.675607 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4221d7500d1458db75926d7b274a654" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.676240 731 tablet_loader.cc:96] loaded metadata for tablet 5e4678ca1c8f498c83c6bc2553882120 (table default.Table_e14bd6f5e20e42b1898bce238d4d986c [id=3a0c95fd4bd84b80ad1f5951cbef92e8])
I20250623 08:04:29.676378 32517 raft_consensus.cc:2466] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:29.676762 731 tablet_loader.cc:96] loaded metadata for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352])
I20250623 08:04:29.677553 731 tablet_loader.cc:96] loaded metadata for tablet 5fdc9e53da264a5ca1f70e1846c906f0 (table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc])
I20250623 08:04:29.677817 342 tablet_bootstrap.cc:654] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.680135 32656 raft_consensus.cc:3058] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.685755 896 consensus_queue.cc:1035] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.696563 746 consensus_queue.cc:1035] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.702966 731 tablet_loader.cc:96] loaded metadata for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5])
I20250623 08:04:29.704200 731 tablet_loader.cc:96] loaded metadata for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 (table default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c [id=387b3fc3d26b457099bcbb3b8ef3ac9b])
I20250623 08:04:29.704679 731 tablet_loader.cc:96] loaded metadata for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0])
I20250623 08:04:29.705077 731 tablet_loader.cc:96] loaded metadata for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 (table default.Table_c2afde03c007450390a0466fe1d39344 [id=77e1411772b54c89a9a8435654249287])
I20250623 08:04:29.706893 731 tablet_loader.cc:96] loaded metadata for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_bc69503004f74bf59fd961b4f209baa6 [id=836b8d720fae4622a556600a1a9c1d91])
I20250623 08:04:29.707325 731 tablet_loader.cc:96] loaded metadata for tablet 71fadb10e6134478b013358bba056389 (table default.Table_4a19c65595654fa98efc42626b6a9d85 [id=2979e4296458439ea5753b6ba67634ac])
I20250623 08:04:29.707718 731 tablet_loader.cc:96] loaded metadata for tablet 724d662bcf754c74a8f3dff91ee258ae (table default.Table_9fc75fd22c6a467782fe3b4c3b123602 [id=1a49c0c47b6647ef8fc4c7c20ad9bc1b])
I20250623 08:04:29.716377 731 tablet_loader.cc:96] loaded metadata for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517])
I20250623 08:04:29.716912 731 tablet_loader.cc:96] loaded metadata for tablet 7419cb91360f49e1b3cb7e90732558c5 (table default.Table_1da3c8ca68084386b7f0d95a6f85927a [id=7cc0919082d04fc4a02aa856a42e5529])
I20250623 08:04:29.717342 731 tablet_loader.cc:96] loaded metadata for tablet 74842fde5152419ca35f3950e465247a (table default.Table_af3136dbadfb43c892a197cc3a5d684e [id=2bfacaaab9f347b1a77e5eb29a9c485e])
I20250623 08:04:29.717824 731 tablet_loader.cc:96] loaded metadata for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0])
I20250623 08:04:29.719666 342 tablet_bootstrap.cc:492] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.719916 342 ts_tablet_manager.cc:1397] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.004s
I20250623 08:04:29.720994 342 raft_consensus.cc:357] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.721293 342 raft_consensus.cc:383] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.721391 342 raft_consensus.cc:738] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.721715 342 consensus_queue.cc:260] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.722374 342 ts_tablet_manager.cc:1428] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:29.723675 731 tablet_loader.cc:96] loaded metadata for tablet 776b412b3f564a7581c6a5e1c84e8499 (table default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652])
I20250623 08:04:29.723867 343 tablet_bootstrap.cc:492] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.724036 343 ts_tablet_manager.cc:1397] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.012s
I20250623 08:04:29.724815 343 raft_consensus.cc:357] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.725024 343 raft_consensus.cc:383] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.725106 343 raft_consensus.cc:738] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.725404 343 consensus_queue.cc:260] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.728683 343 ts_tablet_manager.cc:1428] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:29.733601 731 tablet_loader.cc:96] loaded metadata for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_a615b39019e1478aa69bd46fc2e4a4a9 [id=1e194b3160b64acabac38acfd50a64b3])
I20250623 08:04:29.735283 731 tablet_loader.cc:96] loaded metadata for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_7e39ecbac4a8433e8ae98b0db22c30ea [id=8b20894a91c54a3eb08d5e1b4a42d269])
I20250623 08:04:29.735455 343 tablet_bootstrap.cc:492] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.736114 657 consensus_queue.cc:1035] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250623 08:04:29.736454 731 tablet_loader.cc:96] loaded metadata for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 (table default.Table_3ef2e86e99164cce8e882e6954e2ec17 [id=dbbd669a72954d1aaf169b5870a85fc3])
I20250623 08:04:29.736639 342 tablet_bootstrap.cc:492] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.736955 731 tablet_loader.cc:96] loaded metadata for tablet 965f1f0a107b4653b6aac01e0d8e259a (table default.Table_5f159a56c60144b99a6847ec6080623e [id=ec1f9e35e20c47fcbe2d4b80c7abeabc])
I20250623 08:04:29.739452 731 tablet_loader.cc:96] loaded metadata for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_9f3d2a1f5b3c41fc888725e2b7d45e0e [id=25eb4e3f3a6e42e988d2a3613ef4575f])
I20250623 08:04:29.740273 342 tablet_bootstrap.cc:654] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.740285 731 tablet_loader.cc:96] loaded metadata for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524])
I20250623 08:04:29.740835 731 tablet_loader.cc:96] loaded metadata for tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591])
I20250623 08:04:29.742017 731 tablet_loader.cc:96] loaded metadata for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 (table default.Table_4550e74cc6e845719c1466d3acd1b723 [id=587d6896c7dd40838ce8ad01026379a8])
I20250623 08:04:29.742478 731 tablet_loader.cc:96] loaded metadata for tablet a9ea74d20c5f4eefbcb8508ed1774026 (table default.Table_e851e10456a04a51814a556167f4777f [id=ced0b4cf62924dbfadad02109349bea0])
I20250623 08:04:29.742928 731 tablet_loader.cc:96] loaded metadata for tablet aa398354a9e44633857a75c0bbba08b3 (table default.Table_6bb7361aabf7440589a3958da502f36c [id=150ddf7cf1b141bc8586b94cf6f34aba])
I20250623 08:04:29.743655 731 tablet_loader.cc:96] loaded metadata for tablet adc3cf4355c24b7395fcb22af8520346 (table default.Table_8ef30265c1894796843272f538aaa79e [id=ad13fb2c08c448bebe3e65b0143d66bc])
I20250623 08:04:29.744020 343 tablet_bootstrap.cc:654] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.744797 731 tablet_loader.cc:96] loaded metadata for tablet b19e5f2359a84c7ea02a0b25240a54cb (table default.Table_640be163e2114433bd7079aa3a602116 [id=d422f6202d1842e998a23ec064757334])
I20250623 08:04:29.710197 610 raft_consensus.cc:513] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.747109 731 tablet_loader.cc:96] loaded metadata for tablet b80a91ae2c084020b5d3444b570b28f2 (table default.Table_d226f9dab24f45ae896579983b3d1a67 [id=e0cd5a7f63b443c2a3e86ec75a1e3778])
I20250623 08:04:29.715763 339 tablet_bootstrap.cc:492] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.747465 339 ts_tablet_manager.cc:1397] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.129s user 0.002s sys 0.005s
I20250623 08:04:29.747748 731 tablet_loader.cc:96] loaded metadata for tablet b81263a9ee9d4da1ac721c3b2bd21248 (table default.Table_4e377b569121400d82096403060f9cda [id=49bfa95de97b443ba51225f84820fcf2])
I20250623 08:04:29.748294 339 raft_consensus.cc:357] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.748497 339 raft_consensus.cc:383] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.748610 339 raft_consensus.cc:738] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.748896 339 consensus_queue.cc:260] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.749257 731 tablet_loader.cc:96] loaded metadata for tablet ba3d2fa01d574dd992dc904ac024ba3a (table default.Table_4acf3aff222e477a885bc37d00623ddf [id=6c57db68f5ed45a3b64d1e3a02ce8a2a])
I20250623 08:04:29.749491 339 ts_tablet_manager.cc:1428] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:29.749804 339 tablet_bootstrap.cc:492] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.750391 731 tablet_loader.cc:96] loaded metadata for tablet bfd3e263464c4f2c8af9262acf211193 (table default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 [id=9f97f6deca194137a353ea221d0e3676])
I20250623 08:04:29.725605 685 raft_consensus.cc:491] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.750936 731 tablet_loader.cc:96] loaded metadata for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8])
I20250623 08:04:29.750949 685 raft_consensus.cc:513] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.751543 685 leader_election.cc:290] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.751546 731 tablet_loader.cc:96] loaded metadata for tablet c0f87fd62f1545089d00757f8214593a (table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d])
I20250623 08:04:29.752045 731 tablet_loader.cc:96] loaded metadata for tablet c1596e80225041099fb513ce477eae33 (table default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9])
I20250623 08:04:29.752473 731 tablet_loader.cc:96] loaded metadata for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_a6ef1bf66cd24835896dabd3af080a6a [id=33768fa88bfc4cfba1359030af5e77c9])
I20250623 08:04:29.754297 731 tablet_loader.cc:96] loaded metadata for tablet c7fcf583064f4538913d9157e93068dd (table default.Table_c4287575a32b4847ba2379ef60074ba9 [id=abda89a30971466ea9a1f410d424ef58])
I20250623 08:04:29.755093 731 tablet_loader.cc:96] loaded metadata for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_fe75261182e243ebbbcfbf6e636ba134 [id=5d4bcecfd3904412b0aa6fab7c75d584])
I20250623 08:04:29.755272 339 tablet_bootstrap.cc:654] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.755568 657 consensus_queue.cc:1035] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.755990 731 tablet_loader.cc:96] loaded metadata for tablet cdec6e9573b041e0a57d1e0e0239d97f (table default.Table_48cbdc58fd034e6390087ed7c6346d24 [id=ef1a8122b8044388b4ae24a03da26837])
I20250623 08:04:29.756228 685 consensus_queue.cc:1035] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.756381 610 leader_election.cc:290] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.757166 788 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d31835ec4ad74332a00998d431492466)
I20250623 08:04:29.757181 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67c44697ecab4dd9b43f544a602897ca" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.757328 788 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:29.757493 32517 raft_consensus.cc:2466] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:29.757754 731 tablet_loader.cc:96] loaded metadata for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00])
I20250623 08:04:29.758054 788 leader_election.cc:290] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313)
I20250623 08:04:29.758158 731 tablet_loader.cc:96] loaded metadata for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 (table default.Table_d3f72251faee462c9c301b8aff464bb5 [id=b7671dfa27be46bb8b811b12016434e2])
I20250623 08:04:29.759626 731 tablet_loader.cc:96] loaded metadata for tablet d6ed4cb252b1452087f8ee53ec78d985 (table default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4])
I20250623 08:04:29.759986 731 tablet_loader.cc:96] loaded metadata for tablet d98654c29b9c4e75af6a213cafd4c2c9 (table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa])
I20250623 08:04:29.760617 731 tablet_loader.cc:96] loaded metadata for tablet dc0dad440b494018bc0d6cfd40940bc1 (table default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5])
I20250623 08:04:29.761700 731 tablet_loader.cc:96] loaded metadata for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc])
I20250623 08:04:29.761961 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67c44697ecab4dd9b43f544a602897ca" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.762773 731 tablet_loader.cc:96] loaded metadata for tablet e9f8aa424bbb4aa8a907259270d46656 (table default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e])
I20250623 08:04:29.763522 731 tablet_loader.cc:96] loaded metadata for tablet ecb422550efd40cf82bc92fca5c25eda (table default.Table_de3536331f614b4a8de796d3328c9fa2 [id=5545b396fc5746469e3a8908b0178a4c])
I20250623 08:04:29.764091 731 tablet_loader.cc:96] loaded metadata for tablet eef663ffdaf141aea5e6269b6b2e1c89 (table default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d])
I20250623 08:04:29.764453 731 tablet_loader.cc:96] loaded metadata for tablet effd4fff43fe4d4fb36458986e1dfce1 (table default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac [id=793edd0cb585499197d2f0e57a83be84])
I20250623 08:04:29.765950 731 tablet_loader.cc:96] loaded metadata for tablet f02f36d6c1174a23a1f1043a431d0b6b (table default.Table_75451c4863a34537a328617c53d46cae [id=be88c6ecf20d4ab1b3757c27ce998f87])
I20250623 08:04:29.766611 731 tablet_loader.cc:96] loaded metadata for tablet f279b63b55c64e548ded27b02025c9f5 (table default.Table_6e40e423312e4f9e95174f277b8897ce [id=7038d3807ece414f8a8e76bfe107c3b3])
I20250623 08:04:29.767194 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4221d7500d1458db75926d7b274a654" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.767432 32517 raft_consensus.cc:3058] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.772070 819 consensus_queue.cc:1035] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.772461 32517 raft_consensus.cc:2466] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.773092 731 tablet_loader.cc:96] loaded metadata for tablet f3c850021e914530bb106ab7166be7c9 (table default.Table_fd49e04fcac14f0194cc9fed910601fc [id=17b055700da64cc1b19f93dd5edf40d5])
I20250623 08:04:29.773700 731 tablet_loader.cc:96] loaded metadata for tablet f3d32798168345edb9898751d083057c (table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a])
I20250623 08:04:29.774188 731 tablet_loader.cc:96] loaded metadata for tablet f44fd3c67185484e93450682364dc8a8 (table default.Table_e2502c251f3c4a7f8caa4faab352ebda [id=664b85653cd341a6a6afe7d04fb9c2ac])
I20250623 08:04:29.774704 731 tablet_loader.cc:96] loaded metadata for tablet f4a0b9402a264800a1ea39707c10da0c (table default.Table_244bcdc8d69b4f61a337dac403ffe6cb [id=b3ab55b66d754642aed2f746d9a35807])
I20250623 08:04:29.775460 731 tablet_loader.cc:96] loaded metadata for tablet f7d69bcad8684d99a0b08eefb470ab01 (table default.Table_0bb2584ec3e0467194a9591bbe5ae477 [id=87e7893d85264d4c9cbcc5564f840390])
I20250623 08:04:29.775646 610 consensus_queue.cc:1035] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.777000 731 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 08:04:29.777073 32069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 4 candidate_status { last_received { term: 3 index: 357 } } ignore_live_leader: false dest_uuid: "d31835ec4ad74332a00998d431492466" is_pre_election: true
W20250623 08:04:29.776940 32314 leader_election.cc:343] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.779135 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4221d7500d1458db75926d7b274a654" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.779594 32654 raft_consensus.cc:3058] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.782547 32314 leader_election.cc:304] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:29.783396 610 raft_consensus.cc:2802] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.783495 610 raft_consensus.cc:491] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.783600 610 raft_consensus.cc:3058] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.787711 32654 raft_consensus.cc:2466] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.787691 610 raft_consensus.cc:513] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.793820 731 catalog_manager.cc:1261] Loaded cluster ID: bde910f56f35432e94ef19d006edb93c
I20250623 08:04:29.793951 731 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 08:04:29.788234 32314 leader_election.cc:304] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.795801 685 raft_consensus.cc:2802] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.795989 685 raft_consensus.cc:695] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:29.796284 685 consensus_queue.cc:237] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.807044 610 leader_election.cc:290] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.808079 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67c44697ecab4dd9b43f544a602897ca" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.808351 32518 raft_consensus.cc:3058] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.812330 32518 raft_consensus.cc:2466] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.813478 731 catalog_manager.cc:1506] Loading token signing keys...
I20250623 08:04:29.817965 32314 leader_election.cc:304] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.818428 610 raft_consensus.cc:2802] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.818759 610 raft_consensus.cc:695] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:29.819092 610 consensus_queue.cc:237] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.820500 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67c44697ecab4dd9b43f544a602897ca" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
W20250623 08:04:29.821142 32314 leader_election.cc:343] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.825019 731 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Loaded TSK: 0
I20250623 08:04:29.826160 731 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250623 08:04:29.849092 391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111)
I20250623 08:04:29.849640 896 consensus_queue.cc:1035] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.856583 710 consensus_queue.cc:1035] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.873502 710 consensus_queue.cc:1035] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.875922 343 tablet_bootstrap.cc:492] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.876121 343 ts_tablet_manager.cc:1397] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.141s user 0.025s sys 0.017s
I20250623 08:04:29.877081 343 raft_consensus.cc:357] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.877328 343 raft_consensus.cc:383] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.877449 343 raft_consensus.cc:738] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.877763 343 consensus_queue.cc:260] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.878826 343 ts_tablet_manager.cc:1428] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250623 08:04:29.880116 343 tablet_bootstrap.cc:492] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.882332 339 tablet_bootstrap.cc:492] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.882512 339 ts_tablet_manager.cc:1397] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.133s user 0.009s sys 0.000s
I20250623 08:04:29.883025 343 tablet_bootstrap.cc:654] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.884783 710 consensus_queue.cc:1035] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.889084 339 raft_consensus.cc:357] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.889350 339 raft_consensus.cc:383] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.889442 339 raft_consensus.cc:738] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.889730 339 consensus_queue.cc:260] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.891135 339 ts_tablet_manager.cc:1428] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250623 08:04:29.891467 339 tablet_bootstrap.cc:492] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:29.893425 339 tablet_bootstrap.cc:654] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.894351 874 raft_consensus.cc:491] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.895442 874 raft_consensus.cc:513] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.896154 874 leader_election.cc:290] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.898809 874 raft_consensus.cc:491] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.898972 874 raft_consensus.cc:513] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.899637 874 leader_election.cc:290] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.901038 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b81263a9ee9d4da1ac721c3b2bd21248" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:29.901758 32453 leader_election.cc:343] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.902710 343 tablet_bootstrap.cc:492] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.902910 343 ts_tablet_manager.cc:1397] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.003s
I20250623 08:04:29.906302 343 raft_consensus.cc:357] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.906580 343 raft_consensus.cc:383] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.906677 343 raft_consensus.cc:738] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.908768 32058 master_service.cc:948] ReplaceTablet: received request to replace tablet 4fe30079045d437aa61bb5bfbfb96133 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:29.909309 819 raft_consensus.cc:491] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.909497 819 raft_consensus.cc:513] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.910293 819 leader_election.cc:290] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.911865 343 consensus_queue.cc:260] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.912024 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e47b10f46b8402e94354ebca573c194" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.912259 32055 master_service.cc:948] ReplaceTablet: received request to replace tablet c0f87fd62f1545089d00757f8214593a from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:29.913290 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_2092a828aea549a38d1bd42248c8804c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:29.913852 874 raft_consensus.cc:491] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.914028 874 raft_consensus.cc:513] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.913364 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_2b1942032adc43d78d1f9e83417b42eb" } new_table_name: "default.Table_06b64498b6e148d5b86187f2ce7e4c31" modify_external_catalogs: true
I20250623 08:04:29.914685 874 leader_election.cc:290] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:29.915485 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2092a828aea549a38d1bd42248c8804c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:29.915588 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "432d1642c0e44119b5bc05858463a136" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:29.916610 32453 leader_election.cc:343] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.917685 874 consensus_queue.cc:1035] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.918368 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e47b10f46b8402e94354ebca573c194" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.918649 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "432d1642c0e44119b5bc05858463a136" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.918825 32380 raft_consensus.cc:2391] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:29.918886 32654 raft_consensus.cc:2466] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:29.919276 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_168a55a099724943a6e452febfeae7b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:29.919371 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_c64abc5f2e1240bba6400c28a19631a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:29.920104 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c64abc5f2e1240bba6400c28a19631a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:29.920132 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_168a55a099724943a6e452febfeae7b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:29.921247 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_e216bbb5ac2a4720b36183e30bd6ed91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:29.921890 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e216bbb5ac2a4720b36183e30bd6ed91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:29.921895 746 raft_consensus.cc:491] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.922027 746 raft_consensus.cc:513] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.922274 343 ts_tablet_manager.cc:1428] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250623 08:04:29.922775 746 leader_election.cc:290] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.923007 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_0666030ad7ec4ef89335cdbaf0cca443"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:29.923466 343 tablet_bootstrap.cc:492] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.923485 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "279624b8bb834b95a4f08045968ebfb4" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:29.923581 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0666030ad7ec4ef89335cdbaf0cca443 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:29.923660 32517 raft_consensus.cc:2466] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:29.924058 32314 leader_election.cc:304] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.924371 746 raft_consensus.cc:2802] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.924495 746 raft_consensus.cc:491] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.924587 746 raft_consensus.cc:3058] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.926653 343 tablet_bootstrap.cc:654] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.927657 746 raft_consensus.cc:513] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.928368 746 leader_election.cc:290] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.928856 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "279624b8bb834b95a4f08045968ebfb4" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:29.928969 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "279624b8bb834b95a4f08045968ebfb4" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:29.929039 32517 raft_consensus.cc:3058] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.929157 32655 raft_consensus.cc:3058] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.930719 32057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fe75261182e243ebbbcfbf6e636ba134" } new_table_name: "default.Table_38934e8ef2e34a6a94c27c2058a1803f" modify_external_catalogs: true
I20250623 08:04:29.931871 32517 raft_consensus.cc:2466] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.932050 32655 raft_consensus.cc:2466] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:29.932180 32702 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:29.932266 32314 leader_election.cc:304] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:29.932596 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "279624b8bb834b95a4f08045968ebfb4" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.932801 32654 raft_consensus.cc:2374] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1. Re-sending same reply.
I20250623 08:04:29.934062 896 raft_consensus.cc:2802] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:29.934243 896 raft_consensus.cc:695] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:29.934671 896 consensus_queue.cc:237] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.937731 32589 leader_election.cc:304] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:29.938056 819 raft_consensus.cc:3058] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.938659 343 tablet_bootstrap.cc:492] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:29.938844 343 ts_tablet_manager.cc:1397] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250623 08:04:29.940018 343 raft_consensus.cc:357] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.940255 343 raft_consensus.cc:383] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.940356 343 raft_consensus.cc:738] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.940634 343 consensus_queue.cc:260] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.941855 819 raft_consensus.cc:2747] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:29.944499 343 ts_tablet_manager.cc:1428] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:29.946621 343 tablet_bootstrap.cc:492] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:29.951188 819 raft_consensus.cc:491] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.951370 819 raft_consensus.cc:513] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.952680 710 consensus_queue.cc:1035] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.952903 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26f4a1b472e848ccb9a1ec71c1dd3516" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.953087 819 leader_election.cc:290] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:29.953190 32380 raft_consensus.cc:2391] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:29.963201 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b81263a9ee9d4da1ac721c3b2bd21248" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.964624 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2060a643d0da47efa868f52bdb331b1f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:29.965305 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2060a643d0da47efa868f52bdb331b1f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:29.970893 339 tablet_bootstrap.cc:492] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:29.971266 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:29.971669 342 tablet_bootstrap.cc:492] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:29.971812 342 ts_tablet_manager.cc:1397] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.249s user 0.001s sys 0.057s
I20250623 08:04:29.972769 342 raft_consensus.cc:357] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.972978 342 raft_consensus.cc:383] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.973100 342 raft_consensus.cc:738] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.973358 342 consensus_queue.cc:260] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.973534 339 ts_tablet_manager.cc:1397] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.001s
I20250623 08:04:29.973923 342 ts_tablet_manager.cc:1428] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:29.974265 342 tablet_bootstrap.cc:492] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:29.976241 32453 leader_election.cc:304] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:29.976991 339 raft_consensus.cc:357] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.977174 339 raft_consensus.cc:383] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:29.977270 339 raft_consensus.cc:738] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:29.977494 339 consensus_queue.cc:260] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:29.980901 699 raft_consensus.cc:2802] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:29.981024 699 raft_consensus.cc:491] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:29.981117 699 raft_consensus.cc:3058] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:29.983318 896 consensus_queue.cc:1035] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:29.984189 699 raft_consensus.cc:513] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.976365 342 tablet_bootstrap.cc:654] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:29.984732 699 leader_election.cc:290] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:29.986234 32453 leader_election.cc:343] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:29.987413 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "432d1642c0e44119b5bc05858463a136" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:29.990589 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26f4a1b472e848ccb9a1ec71c1dd3516" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:29.990882 819 raft_consensus.cc:491] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.990680 848 raft_consensus.cc:491] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:29.991199 848 raft_consensus.cc:513] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.991729 848 leader_election.cc:290] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:29.992453 819 raft_consensus.cc:513] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:29.993072 819 leader_election.cc:290] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:29.993202 339 ts_tablet_manager.cc:1428] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250623 08:04:29.993788 343 tablet_bootstrap.cc:654] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.001796 343 tablet_bootstrap.cc:492] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.001976 343 ts_tablet_manager.cc:1397] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.002s
I20250623 08:04:30.002799 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "432d1642c0e44119b5bc05858463a136" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:30.002856 343 raft_consensus.cc:357] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.002996 32656 raft_consensus.cc:3058] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.003034 343 raft_consensus.cc:383] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.003108 343 raft_consensus.cc:738] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.003350 343 consensus_queue.cc:260] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.003866 343 ts_tablet_manager.cc:1428] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20250623 08:04:30.005756 32453 leader_election.cc:343] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.006338 32453 leader_election.cc:304] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.006695 874 raft_consensus.cc:2747] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:30.006906 343 tablet_bootstrap.cc:492] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:30.010860 32453 leader_election.cc:343] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:30.010874 391 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.61:33313, user_credentials={real_user=slave}} blocked reactor thread for 161825us
I20250623 08:04:30.011154 343 tablet_bootstrap.cc:654] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.012420 32453 leader_election.cc:304] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
W20250623 08:04:30.013073 391 leader_election.cc:336] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313): Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111)
I20250623 08:04:30.013255 391 leader_election.cc:304] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678; no voters: 9752155c20ee42fbb7ff7b7d5b682444, d31835ec4ad74332a00998d431492466
I20250623 08:04:30.013566 788 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 08:04:30.016911 874 raft_consensus.cc:2747] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:30.019239 343 tablet_bootstrap.cc:492] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.019402 343 ts_tablet_manager.cc:1397] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
W20250623 08:04:30.019845 32453 leader_election.cc:343] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.020140 343 raft_consensus.cc:357] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.020320 343 raft_consensus.cc:383] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.020401 343 raft_consensus.cc:738] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.020632 343 consensus_queue.cc:260] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.021507 343 ts_tablet_manager.cc:1428] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:30.022089 343 tablet_bootstrap.cc:492] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.006362 32656 raft_consensus.cc:2466] T 432d1642c0e44119b5bc05858463a136 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:30.024317 343 tablet_bootstrap.cc:654] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.026595 819 consensus_queue.cc:1035] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.027534 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31a67b70cbc64bb3bfd5a34865109a01" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.028281 32589 leader_election.cc:304] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.028645 819 raft_consensus.cc:3058] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.032286 819 raft_consensus.cc:2747] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:30.033895 343 tablet_bootstrap.cc:492] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.034078 343 ts_tablet_manager.cc:1397] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20250623 08:04:30.035712 343 raft_consensus.cc:357] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.036065 343 raft_consensus.cc:383] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.036159 343 raft_consensus.cc:738] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.036491 343 consensus_queue.cc:260] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.037463 343 ts_tablet_manager.cc:1428] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250623 08:04:30.037858 343 tablet_bootstrap.cc:492] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.039314 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet ae7c2ac546114a819a543f1667fb7844 with delete_type TABLET_DATA_DELETED (replaced by tablet d98654c29b9c4e75af6a213cafd4c2c9) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.039683 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7da84969a6464b5caeaf47af0cbaaa4d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:30.042325 32653 raft_consensus.cc:3058] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.042760 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7da84969a6464b5caeaf47af0cbaaa4d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.042851 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31a67b70cbc64bb3bfd5a34865109a01" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.047457 343 tablet_bootstrap.cc:654] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.050791 896 consensus_queue.cc:1035] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.052763 710 consensus_queue.cc:1035] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.057426 339 tablet_bootstrap.cc:492] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:30.059260 343 tablet_bootstrap.cc:492] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.059402 343 ts_tablet_manager.cc:1397] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20250623 08:04:30.060174 343 raft_consensus.cc:357] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.060379 343 raft_consensus.cc:383] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.060626 343 raft_consensus.cc:738] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.060822 343 consensus_queue.cc:260] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.061391 343 ts_tablet_manager.cc:1428] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:30.061661 343 tablet_bootstrap.cc:492] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.064393 343 tablet_bootstrap.cc:654] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.072523 343 tablet_bootstrap.cc:492] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.072698 343 ts_tablet_manager.cc:1397] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250623 08:04:30.073068 746 raft_consensus.cc:491] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.073671 746 raft_consensus.cc:513] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.073671 343 raft_consensus.cc:357] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.073925 343 raft_consensus.cc:383] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.074016 343 raft_consensus.cc:738] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.074256 32589 leader_election.cc:304] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.074347 343 consensus_queue.cc:260] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.074852 944 raft_consensus.cc:2747] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:30.075186 746 leader_election.cc:290] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.075266 343 ts_tablet_manager.cc:1428] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:30.076113 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19536c16ff32460e9db571e4ce411bab" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:30.077459 343 tablet_bootstrap.cc:492] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.079244 343 tablet_bootstrap.cc:654] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.082616 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19536c16ff32460e9db571e4ce411bab" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.082886 32653 raft_consensus.cc:2466] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:30.083377 32314 leader_election.cc:304] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.083676 746 raft_consensus.cc:2802] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.083861 746 raft_consensus.cc:491] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.083997 746 raft_consensus.cc:3058] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.085152 944 raft_consensus.cc:491] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.085317 944 raft_consensus.cc:513] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.086001 944 leader_election.cc:290] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.086652 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba3d2fa01d574dd992dc904ac024ba3a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.087076 746 raft_consensus.cc:513] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.087775 746 leader_election.cc:290] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.088227 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19536c16ff32460e9db571e4ce411bab" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:30.088467 32656 raft_consensus.cc:3058] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.092068 32656 raft_consensus.cc:2466] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:30.092515 32314 leader_election.cc:304] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.092801 746 raft_consensus.cc:2802] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:30.093138 746 raft_consensus.cc:695] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:30.093502 746 consensus_queue.cc:237] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.094884 343 tablet_bootstrap.cc:492] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.095032 343 ts_tablet_manager.cc:1397] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20250623 08:04:30.096783 874 raft_consensus.cc:491] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.097069 32519 raft_consensus.cc:2466] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:30.097880 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19536c16ff32460e9db571e4ce411bab" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:30.098133 32520 raft_consensus.cc:3058] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.098286 343 raft_consensus.cc:357] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.098469 343 raft_consensus.cc:383] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.098546 343 raft_consensus.cc:738] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.100601 343 consensus_queue.cc:260] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.100900 874 raft_consensus.cc:513] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.101164 343 ts_tablet_manager.cc:1428] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:30.101457 343 tablet_bootstrap.cc:492] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.101483 874 leader_election.cc:290] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:30.102411 32453 leader_election.cc:343] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.103277 874 raft_consensus.cc:491] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.103425 874 raft_consensus.cc:513] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.103962 874 leader_election.cc:290] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.104609 32453 leader_election.cc:304] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.104884 874 raft_consensus.cc:2802] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:30.105031 874 raft_consensus.cc:695] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:30.105266 874 consensus_queue.cc:237] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.108143 339 tablet_bootstrap.cc:654] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.108716 343 tablet_bootstrap.cc:654] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.111152 32520 raft_consensus.cc:2466] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:30.111696 699 consensus_queue.cc:1035] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:30.112017 32453 leader_election.cc:343] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.116914 32453 leader_election.cc:304] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.117256 699 raft_consensus.cc:2747] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:30.118526 699 consensus_queue.cc:1035] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.121656 944 consensus_queue.cc:1035] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.123363 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7419cb91360f49e1b3cb7e90732558c5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.123615 32656 raft_consensus.cc:2466] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:30.123914 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7419cb91360f49e1b3cb7e90732558c5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.124218 32379 raft_consensus.cc:2466] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:30.124593 342 tablet_bootstrap.cc:492] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:30.124773 342 ts_tablet_manager.cc:1397] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.151s user 0.020s sys 0.016s
I20250623 08:04:30.125629 342 raft_consensus.cc:357] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.125877 342 raft_consensus.cc:383] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.125972 342 raft_consensus.cc:738] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.126168 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74842fde5152419ca35f3950e465247a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.126204 342 consensus_queue.cc:260] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.126356 32656 raft_consensus.cc:2466] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:30.126912 342 ts_tablet_manager.cc:1428] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250623 08:04:30.127568 342 tablet_bootstrap.cc:492] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:30.129415 342 tablet_bootstrap.cc:654] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.129606 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74842fde5152419ca35f3950e465247a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.129859 952 tablet_replica.cc:331] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:30.141058 339 tablet_bootstrap.cc:492] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:30.141263 339 ts_tablet_manager.cc:1397] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.002s
I20250623 08:04:30.142287 339 raft_consensus.cc:357] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.142642 339 raft_consensus.cc:383] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.144682 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 4fe30079045d437aa61bb5bfbfb96133
I20250623 08:04:30.146157 32058 catalog_manager.cc:6652] ReplaceTablet: tablet 4fe30079045d437aa61bb5bfbfb96133 deleted and replaced by tablet 18ea63263b9b466e922169eecafab747
I20250623 08:04:30.160846 710 consensus_queue.cc:1035] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.167199 710 raft_consensus.cc:491] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.167398 710 raft_consensus.cc:513] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.168219 710 leader_election.cc:290] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.176283 32055 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet c0f87fd62f1545089d00757f8214593a
I20250623 08:04:30.176884 32055 catalog_manager.cc:6652] ReplaceTablet: tablet c0f87fd62f1545089d00757f8214593a deleted and replaced by tablet 1f76ef24c69f4b17a373b242d71da8b9
I20250623 08:04:30.177165 32055 master_service.cc:948] ReplaceTablet: received request to replace tablet 4ac9f38877b24d619c280a1aaa0b7b08 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:30.183527 343 tablet_bootstrap.cc:492] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.183703 343 ts_tablet_manager.cc:1397] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.082s user 0.019s sys 0.015s
I20250623 08:04:30.184732 343 raft_consensus.cc:357] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.184928 343 raft_consensus.cc:383] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.184998 343 raft_consensus.cc:738] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.185242 343 consensus_queue.cc:260] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.142735 339 raft_consensus.cc:738] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.149231 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:30.169245 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31a67b70cbc64bb3bfd5a34865109a01" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:30.188215 32589 leader_election.cc:343] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.193862 944 consensus_queue.cc:1035] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.203301 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_8ec723069381477ca64f5aaad0df76fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.203917 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ec723069381477ca64f5aaad0df76fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.211440 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe30079045d437aa61bb5bfbfb96133 with delete_type TABLET_DATA_DELETED (replaced by tablet 18ea63263b9b466e922169eecafab747) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.211835 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f87fd62f1545089d00757f8214593a with delete_type TABLET_DATA_DELETED (replaced by tablet 1f76ef24c69f4b17a373b242d71da8b9) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.213899 343 ts_tablet_manager.cc:1428] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20250623 08:04:30.214300 343 tablet_bootstrap.cc:492] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.215662 32051 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_5f159a56c60144b99a6847ec6080623e" } modify_external_catalogs: true
I20250623 08:04:30.215893 32051 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_5f159a56c60144b99a6847ec6080623e" } modify_external_catalogs: true
I20250623 08:04:30.216440 343 tablet_bootstrap.cc:654] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.218165 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_f40b881c21294c10b35962b75a30524c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.218739 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f40b881c21294c10b35962b75a30524c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.187129 339 consensus_queue.cc:260] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.223922 343 tablet_bootstrap.cc:492] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.224130 343 ts_tablet_manager.cc:1397] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250623 08:04:30.225075 343 raft_consensus.cc:357] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.225288 343 raft_consensus.cc:383] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.225418 343 raft_consensus.cc:738] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.226617 979 tablet_replica.cc:331] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:30.227120 979 raft_consensus.cc:2241] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:30.227710 979 raft_consensus.cc:2270] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.227802 343 consensus_queue.cc:260] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.228647 339 ts_tablet_manager.cc:1428] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.087s user 0.000s sys 0.003s
I20250623 08:04:30.228996 339 tablet_bootstrap.cc:492] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:30.229233 343 ts_tablet_manager.cc:1428] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250623 08:04:30.229521 343 tablet_bootstrap.cc:492] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.230794 979 ts_tablet_manager.cc:1905] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.232604 343 tablet_bootstrap.cc:654] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.233076 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet c6a97b34e386443fa19a9601621b5017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.233330 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe30079045d437aa61bb5bfbfb96133 with delete_type TABLET_DATA_DELETED (replaced by tablet 18ea63263b9b466e922169eecafab747) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.236604 952 raft_consensus.cc:2241] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:30.239565 343 tablet_bootstrap.cc:492] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.239733 343 ts_tablet_manager.cc:1397] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250623 08:04:30.240424 343 raft_consensus.cc:357] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.240628 343 raft_consensus.cc:383] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.240705 343 raft_consensus.cc:738] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.240929 343 consensus_queue.cc:260] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.243407 343 ts_tablet_manager.cc:1428] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:30.243705 343 tablet_bootstrap.cc:492] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.245299 343 tablet_bootstrap.cc:654] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.248329 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f87fd62f1545089d00757f8214593a with delete_type TABLET_DATA_DELETED (replaced by tablet 1f76ef24c69f4b17a373b242d71da8b9) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.250541 343 tablet_bootstrap.cc:492] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.250692 343 ts_tablet_manager.cc:1397] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250623 08:04:30.251942 952 raft_consensus.cc:2270] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.252334 343 raft_consensus.cc:357] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.252506 343 raft_consensus.cc:383] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.252588 343 raft_consensus.cc:738] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.253988 343 consensus_queue.cc:260] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.254992 343 ts_tablet_manager.cc:1428] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250623 08:04:30.255272 343 tablet_bootstrap.cc:492] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.257090 343 tablet_bootstrap.cc:654] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.257658 952 ts_tablet_manager.cc:1905] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.258392 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba3d2fa01d574dd992dc904ac024ba3a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:30.258689 32517 raft_consensus.cc:2391] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:30.262379 343 tablet_bootstrap.cc:492] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.262532 343 ts_tablet_manager.cc:1397] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:30.264035 343 raft_consensus.cc:357] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.265152 343 raft_consensus.cc:383] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.265247 343 raft_consensus.cc:738] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.265502 343 consensus_queue.cc:260] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.266181 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31a67b70cbc64bb3bfd5a34865109a01" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:30.266440 32517 raft_consensus.cc:2466] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:30.266790 343 ts_tablet_manager.cc:1428] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:30.267077 343 tablet_bootstrap.cc:492] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.267740 32589 leader_election.cc:304] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.268343 32589 leader_election.cc:304] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.268795 343 tablet_bootstrap.cc:654] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.268936 944 raft_consensus.cc:3058] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.269033 979 ts_tablet_manager.cc:1918] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:30.269259 945 consensus_queue.cc:1035] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.269809 945 raft_consensus.cc:2802] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.269939 945 raft_consensus.cc:491] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.270040 945 raft_consensus.cc:3058] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.271131 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe30079045d437aa61bb5bfbfb96133 with delete_type TABLET_DATA_DELETED (replaced by tablet 18ea63263b9b466e922169eecafab747) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.271289 32453 leader_election.cc:304] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.271348 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f87fd62f1545089d00757f8214593a with delete_type TABLET_DATA_DELETED (replaced by tablet 1f76ef24c69f4b17a373b242d71da8b9) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:30.271593 32453 leader_election.cc:343] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.271873 848 raft_consensus.cc:491] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.271992 657 raft_consensus.cc:2802] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.272105 657 raft_consensus.cc:491] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.272079 848 raft_consensus.cc:513] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.272241 657 raft_consensus.cc:3058] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.272815 848 leader_election.cc:290] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.294824 657 raft_consensus.cc:513] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.303668 32053 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c4287575a32b4847ba2379ef60074ba9" } new_table_name: "default.Table_a65e08da5dbe4b408014e6c3b662450f" modify_external_catalogs: true
I20250623 08:04:30.306339 710 raft_consensus.cc:491] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.306596 710 raft_consensus.cc:513] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.307462 710 leader_election.cc:290] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.308912 819 raft_consensus.cc:491] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.309943 979 log.cc:1199] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/4fe30079045d437aa61bb5bfbfb96133
I20250623 08:04:30.309826 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_2fceb5254f914cd9a952ae0a805150c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.310492 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fceb5254f914cd9a952ae0a805150c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.312868 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_de3536331f614b4a8de796d3328c9fa2" } new_table_name: "default.Table_4c1c98c205e04f4e90e5a301451870da" modify_external_catalogs: true
I20250623 08:04:30.314587 339 tablet_bootstrap.cc:654] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.316265 981 tablet_replica.cc:331] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:30.318895 981 raft_consensus.cc:2241] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:30.319391 981 raft_consensus.cc:2270] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.321702 981 ts_tablet_manager.cc:1905] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.310822 819 raft_consensus.cc:513] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.332394 819 leader_election.cc:290] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.340780 710 raft_consensus.cc:491] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.341241 32348 tablet_service.cc:1468] Processing CreateTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 (DEFAULT_TABLE table=default.Table_d3f72251faee462c9c301b8aff464bb5 [id=b7671dfa27be46bb8b811b12016434e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.341336 945 raft_consensus.cc:513] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.341740 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b2af698b6e4ed0ae66a9bb32be7b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.342903 944 raft_consensus.cc:2747] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:30.343379 945 leader_election.cc:290] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.344887 710 raft_consensus.cc:513] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.345479 710 leader_election.cc:290] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.345691 32622 tablet_service.cc:1468] Processing CreateTablet for tablet e602d391526d4b0eb0dbf5189afbe075 (DEFAULT_TABLE table=default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.346101 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e602d391526d4b0eb0dbf5189afbe075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.346633 32623 tablet_service.cc:1468] Processing CreateTablet for tablet 71fadb10e6134478b013358bba056389 (DEFAULT_TABLE table=default.Table_4a19c65595654fa98efc42626b6a9d85 [id=2979e4296458439ea5753b6ba67634ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.347128 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71fadb10e6134478b013358bba056389. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.350652 342 tablet_bootstrap.cc:492] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:30.350850 342 ts_tablet_manager.cc:1397] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.224s user 0.001s sys 0.003s
I20250623 08:04:30.352406 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 21080ab069d74457a9f7fe3aabdb3c7c (DEFAULT_TABLE table=default.Table_f1d8f7ef7cf144eaa1fd741b10eac3d6 [id=9060c1640fca461c9020722b3165d144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.352800 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21080ab069d74457a9f7fe3aabdb3c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.353478 657 leader_election.cc:290] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.355870 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 736eb56dae194df2af90bf33d3294a4e (DEFAULT_TABLE table=default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:30.357862 32050 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ecb422550efd40cf82bc92fca5c25eda consensus state has no leader
I20250623 08:04:30.358220 32050 master_service.cc:948] ReplaceTablet: received request to replace tablet cde7df7d39a54114a4dcb28cfd1eb429 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:30.359103 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 9fd478c007914d2cb09d929b50322a64 (DEFAULT_TABLE table=default.Table_9f3d2a1f5b3c41fc888725e2b7d45e0e [id=25eb4e3f3a6e42e988d2a3613ef4575f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.365927 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736eb56dae194df2af90bf33d3294a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.366541 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 21080ab069d74457a9f7fe3aabdb3c7c (DEFAULT_TABLE table=default.Table_f1d8f7ef7cf144eaa1fd741b10eac3d6 [id=9060c1640fca461c9020722b3165d144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.367041 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 736eb56dae194df2af90bf33d3294a4e (DEFAULT_TABLE table=default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.367067 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f3d32798168345edb9898751d083057c (table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]). Expected version 1 got 0
I20250623 08:04:30.367177 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 9fd478c007914d2cb09d929b50322a64 (DEFAULT_TABLE table=default.Table_9f3d2a1f5b3c41fc888725e2b7d45e0e [id=25eb4e3f3a6e42e988d2a3613ef4575f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.367717 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21080ab069d74457a9f7fe3aabdb3c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.367703 32059 catalog_manager.cc:5582] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:30.369081 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 1f76ef24c69f4b17a373b242d71da8b9 (DEFAULT_TABLE table=default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.366225 32636 tablet_service.cc:1468] Processing CreateTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 (DEFAULT_TABLE table=default.Table_d3f72251faee462c9c301b8aff464bb5 [id=b7671dfa27be46bb8b811b12016434e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.369446 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f02f36d6c1174a23a1f1043a431d0b6b (table default.Table_75451c4863a34537a328617c53d46cae [id=be88c6ecf20d4ab1b3757c27ce998f87]). Expected version 2 got 0
I20250623 08:04:30.369448 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 3249605158634f4eab5b9d61357c383f (DEFAULT_TABLE table=default.Table_79be3d7247284828a82f02d96faa60cc [id=c618740b8cb040bcb78c452b3d747391]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.370098 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 38ecd4c243b840f4b66d8e01078dca64 (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]). Expected version 1 got 0
I20250623 08:04:30.370271 979 ts_tablet_manager.cc:1939] T 4fe30079045d437aa61bb5bfbfb96133 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:30.370558 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 3249605158634f4eab5b9d61357c383f (DEFAULT_TABLE table=default.Table_79be3d7247284828a82f02d96faa60cc [id=c618740b8cb040bcb78c452b3d747391]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.370779 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fd478c007914d2cb09d929b50322a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.371098 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 71fadb10e6134478b013358bba056389 (DEFAULT_TABLE table=default.Table_4a19c65595654fa98efc42626b6a9d85 [id=2979e4296458439ea5753b6ba67634ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.371160 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 1f76ef24c69f4b17a373b242d71da8b9 (DEFAULT_TABLE table=default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.371556 896 consensus_queue.cc:1035] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250623 08:04:30.373467 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736eb56dae194df2af90bf33d3294a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.373661 32351 tablet_service.cc:1468] Processing CreateTablet for tablet e602d391526d4b0eb0dbf5189afbe075 (DEFAULT_TABLE table=default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.376256 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 5f1fb8231e8e45208ac4f412134d8696 (DEFAULT_TABLE table=default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.376336 342 raft_consensus.cc:357] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.376530 342 raft_consensus.cc:383] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.376618 342 raft_consensus.cc:738] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.376817 342 consensus_queue.cc:260] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.377388 342 ts_tablet_manager.cc:1428] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250623 08:04:30.377709 342 tablet_bootstrap.cc:492] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:30.378859 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3249605158634f4eab5b9d61357c383f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.379062 32624 tablet_service.cc:1468] Processing CreateTablet for tablet 724d662bcf754c74a8f3dff91ee258ae (DEFAULT_TABLE table=default.Table_9fc75fd22c6a467782fe3b4c3b123602 [id=1a49c0c47b6647ef8fc4c7c20ad9bc1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.380230 952 ts_tablet_manager.cc:1918] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:30.380362 952 log.cc:1199] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/ae7c2ac546114a819a543f1667fb7844
I20250623 08:04:30.380801 342 tablet_bootstrap.cc:654] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.381973 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3249605158634f4eab5b9d61357c383f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.382058 952 ts_tablet_manager.cc:1939] T ae7c2ac546114a819a543f1667fb7844 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:30.382859 952 tablet_replica.cc:331] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:30.383122 952 raft_consensus.cc:2241] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:30.383323 952 raft_consensus.cc:2270] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.384285 952 ts_tablet_manager.cc:1905] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.385653 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 5f1fb8231e8e45208ac4f412134d8696 (DEFAULT_TABLE table=default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.389670 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724d662bcf754c74a8f3dff91ee258ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.389886 32358 tablet_service.cc:1468] Processing CreateTablet for tablet aa398354a9e44633857a75c0bbba08b3 (DEFAULT_TABLE table=default.Table_6bb7361aabf7440589a3958da502f36c [id=150ddf7cf1b141bc8586b94cf6f34aba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.390543 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f76ef24c69f4b17a373b242d71da8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.391059 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa398354a9e44633857a75c0bbba08b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.391026 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 724d662bcf754c74a8f3dff91ee258ae (DEFAULT_TABLE table=default.Table_9fc75fd22c6a467782fe3b4c3b123602 [id=1a49c0c47b6647ef8fc4c7c20ad9bc1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.391448 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fd478c007914d2cb09d929b50322a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.391685 896 consensus_queue.cc:1035] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250623 08:04:30.392621 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f76ef24c69f4b17a373b242d71da8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.393764 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f1fb8231e8e45208ac4f412134d8696. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.394016 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e602d391526d4b0eb0dbf5189afbe075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.394656 979 tablet_replica.cc:331] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:30.394688 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b2af698b6e4ed0ae66a9bb32be7b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.396766 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f1fb8231e8e45208ac4f412134d8696. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.406975 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71fadb10e6134478b013358bba056389. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.408785 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724d662bcf754c74a8f3dff91ee258ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.410619 896 consensus_queue.cc:1035] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.381556 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 4fe30079045d437aa61bb5bfbfb96133 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]) successfully deleted
W20250623 08:04:30.387228 32045 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ecb422550efd40cf82bc92fca5c25eda (table default.Table_4c1c98c205e04f4e90e5a301451870da [id=5545b396fc5746469e3a8908b0178a4c], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ecb422550efd40cf82bc92fca5c25eda consensus state has no leader
I20250623 08:04:30.390111 32056 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e962b9945a5041039f7e33b98b653175" } modify_external_catalogs: true
I20250623 08:04:30.414397 32056 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e962b9945a5041039f7e33b98b653175" } modify_external_catalogs: true
I20250623 08:04:30.370397 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:30.416251 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:30.416538 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:30.416977 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:30.423635 32355 tablet_service.cc:1468] Processing CreateTablet for tablet e9f8aa424bbb4aa8a907259270d46656 (DEFAULT_TABLE table=default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.423658 32632 tablet_service.cc:1468] Processing CreateTablet for tablet 52d0a356c74f4daea9093c998b4bf73e (DEFAULT_TABLE table=default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.424111 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f8aa424bbb4aa8a907259270d46656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.424283 32618 tablet_service.cc:1468] Processing CreateTablet for tablet e9f8aa424bbb4aa8a907259270d46656 (DEFAULT_TABLE table=default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.425398 32634 tablet_service.cc:1468] Processing CreateTablet for tablet ecb422550efd40cf82bc92fca5c25eda (DEFAULT_TABLE table=default.Table_4c1c98c205e04f4e90e5a301451870da [id=5545b396fc5746469e3a8908b0178a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.425761 32059 catalog_manager.cc:5582] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:30.426002 32619 tablet_service.cc:1468] Processing CreateTablet for tablet bfd3e263464c4f2c8af9262acf211193 (DEFAULT_TABLE table=default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 [id=9f97f6deca194137a353ea221d0e3676]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.426198 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 0861107ab4624a05bd7a4d24b420c502 (table default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]). Expected version 1 got 0
I20250623 08:04:30.426494 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_a6ef1bf66cd24835896dabd3af080a6a [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 1 got 0
I20250623 08:04:30.426677 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:30.427006 896 raft_consensus.cc:491] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.426859 32059 catalog_manager.cc:5582] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } }
I20250623 08:04:30.427222 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 4ac9f38877b24d619c280a1aaa0b7b08 (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:30.427171 746 raft_consensus.cc:491] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.427170 896 raft_consensus.cc:513] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.427382 746 raft_consensus.cc:513] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.427534 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:30.427856 896 leader_election.cc:290] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:30.427947 746 leader_election.cc:290] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.428236 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]). Expected version 1 got 0
I20250623 08:04:30.428629 32059 catalog_manager.cc:5582] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:30.434410 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 0861e45cf546487d9812caa06e2704eb (DEFAULT_TABLE table=default.Table_1671bbefe7bd4d4b86a917373288a1a6 [id=baaeae53fdcb49b7889d8379fa203c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.434718 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 0311ce67989f49e8866234a88e5b4d09 (table default.Table_e962b9945a5041039f7e33b98b653175 [id=9d9e232100264e378516c06c61310750]). Expected version 1 got 0
I20250623 08:04:30.440871 32059 catalog_manager.cc:5582] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } }
I20250623 08:04:30.442050 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_7e39ecbac4a8433e8ae98b0db22c30ea [id=8b20894a91c54a3eb08d5e1b4a42d269]). Expected version 2 got 0
I20250623 08:04:30.442878 32059 catalog_manager.cc:5582] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:30.443338 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0861e45cf546487d9812caa06e2704eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.443852 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (table default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b]). Expected version 1 got 0
I20250623 08:04:30.444164 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:30.444442 32059 catalog_manager.cc:5582] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:30.445004 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5fdc9e53da264a5ca1f70e1846c906f0 (table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]). Expected version 1 got 0
I20250623 08:04:30.445637 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]). Expected version 2 got 0
I20250623 08:04:30.447341 32059 catalog_manager.cc:5582] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:30.447973 32059 catalog_manager.cc:5582] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:30.448661 32059 catalog_manager.cc:5582] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:30.461632 32356 tablet_service.cc:1468] Processing CreateTablet for tablet dc0dad440b494018bc0d6cfd40940bc1 (DEFAULT_TABLE table=default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.462065 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc0dad440b494018bc0d6cfd40940bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.462785 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 52d0a356c74f4daea9093c998b4bf73e (DEFAULT_TABLE table=default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.463182 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52d0a356c74f4daea9093c998b4bf73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.461477 32349 tablet_service.cc:1468] Processing CreateTablet for tablet bfd3e263464c4f2c8af9262acf211193 (DEFAULT_TABLE table=default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 [id=9f97f6deca194137a353ea221d0e3676]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.468360 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32c82141edea4068b54755a85c6d7fb8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.468585 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32c82141edea4068b54755a85c6d7fb8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:30.468788 32518 raft_consensus.cc:2466] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:30.469069 32629 tablet_service.cc:1468] Processing CreateTablet for tablet dc0dad440b494018bc0d6cfd40940bc1 (DEFAULT_TABLE table=default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.469449 32635 tablet_service.cc:1468] Processing CreateTablet for tablet 18ea63263b9b466e922169eecafab747 (DEFAULT_TABLE table=default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.469766 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 0e9bf7cba2244703b7402cb4be1a005e (DEFAULT_TABLE table=default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.470121 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d02f74c24a174b18b08a870a4e47fefb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.470324 32653 raft_consensus.cc:2466] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:30.470098 32619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfd3e263464c4f2c8af9262acf211193. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.470469 32622 tablet_service.cc:1468] Processing CreateTablet for tablet aa398354a9e44633857a75c0bbba08b3 (DEFAULT_TABLE table=default.Table_6bb7361aabf7440589a3958da502f36c [id=150ddf7cf1b141bc8586b94cf6f34aba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.470885 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9bf7cba2244703b7402cb4be1a005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.470918 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d02f74c24a174b18b08a870a4e47fefb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:30.471094 32517 raft_consensus.cc:2466] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:30.474886 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ea63263b9b466e922169eecafab747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.477136 657 raft_consensus.cc:491] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.477306 657 raft_consensus.cc:513] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.477895 657 leader_election.cc:290] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.478890 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc0dad440b494018bc0d6cfd40940bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.483148 32050 master_service.cc:948] ReplaceTablet: received request to replace tablet 2ecf4af9e476495683bd63cac20d4672 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:30.483745 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_7359680519754448a8464b0e8131bd55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.484302 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7359680519754448a8464b0e8131bd55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.484427 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52d0a356c74f4daea9093c998b4bf73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.484812 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_3115dd6f7dc948aa927f4ab714867992"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.485245 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3115dd6f7dc948aa927f4ab714867992 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.485657 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a94b76793b7a49a687d0052141dd50c4" } new_table_name: "default.Table_a1f91329476d4e4f9be66462bda04dbf" modify_external_catalogs: true
I20250623 08:04:30.486124 32050 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e7f8192c00524b5899152fa3f717b898" } modify_external_catalogs: true
I20250623 08:04:30.486267 32050 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e7f8192c00524b5899152fa3f717b898" } modify_external_catalogs: true
I20250623 08:04:30.489966 32360 tablet_service.cc:1468] Processing CreateTablet for tablet ecb422550efd40cf82bc92fca5c25eda (DEFAULT_TABLE table=default.Table_4c1c98c205e04f4e90e5a301451870da [id=5545b396fc5746469e3a8908b0178a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.490548 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f8aa424bbb4aa8a907259270d46656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.489078 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 0e9bf7cba2244703b7402cb4be1a005e (DEFAULT_TABLE table=default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.494901 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb422550efd40cf82bc92fca5c25eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.495877 32055 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 4ac9f38877b24d619c280a1aaa0b7b08
I20250623 08:04:30.496309 32055 catalog_manager.cc:6652] ReplaceTablet: tablet 4ac9f38877b24d619c280a1aaa0b7b08 deleted and replaced by tablet a8de2b1105564fdab386442b1de4cedb
W20250623 08:04:30.496297 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 77994.9us
I20250623 08:04:30.496667 32055 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c30b48097bd643b18b2a744839780b9e" } modify_external_catalogs: true
I20250623 08:04:30.496845 32055 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c30b48097bd643b18b2a744839780b9e" } modify_external_catalogs: true
I20250623 08:04:30.497303 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_e8864c4ee64b4f4dbf04caa321024840"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.497846 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8864c4ee64b4f4dbf04caa321024840 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.498458 32349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfd3e263464c4f2c8af9262acf211193. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.499505 32619 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ac9f38877b24d619c280a1aaa0b7b08 with delete_type TABLET_DATA_DELETED (replaced by tablet a8de2b1105564fdab386442b1de4cedb) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.489565 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 18ea63263b9b466e922169eecafab747 (DEFAULT_TABLE table=default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.501945 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb422550efd40cf82bc92fca5c25eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.507568 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9bf7cba2244703b7402cb4be1a005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.508546 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa398354a9e44633857a75c0bbba08b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.510826 32050 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a4ca66fa08cc441ab5571f9968f30914" } modify_external_catalogs: true
I20250623 08:04:30.511040 32050 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a4ca66fa08cc441ab5571f9968f30914" } modify_external_catalogs: true
I20250623 08:04:30.511402 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ea63263b9b466e922169eecafab747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.511386 979 raft_consensus.cc:2241] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:30.511575 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_7b9c8fa18d93462685ad015a8cf62225"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:30.513712 32654 raft_consensus.cc:2466] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
W20250623 08:04:30.514223 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b9c8fa18d93462685ad015a8cf62225 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.519831 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_c4f705a7cbd940da941c460b61c5dece"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.520529 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4f705a7cbd940da941c460b61c5dece in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.521175 979 raft_consensus.cc:2270] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.524675 979 ts_tablet_manager.cc:1905] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.520498 32453 leader_election.cc:304] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.525302 32623 tablet_service.cc:1468] Processing CreateTablet for tablet 0861e45cf546487d9812caa06e2704eb (DEFAULT_TABLE table=default.Table_1671bbefe7bd4d4b86a917373288a1a6 [id=baaeae53fdcb49b7889d8379fa203c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.525650 657 raft_consensus.cc:2802] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.525759 657 raft_consensus.cc:491] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.525864 657 raft_consensus.cc:3058] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.528983 657 raft_consensus.cc:513] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.529842 657 leader_election.cc:290] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.529887 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:30.530256 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet f02f36d6c1174a23a1f1043a431d0b6b (table default.Table_75451c4863a34537a328617c53d46cae [id=be88c6ecf20d4ab1b3757c27ce998f87]). Expected version 2 got 0
I20250623 08:04:30.530447 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:30.534041 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0861e45cf546487d9812caa06e2704eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.534461 944 raft_consensus.cc:491] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.535118 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32c82141edea4068b54755a85c6d7fb8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.535343 32654 raft_consensus.cc:2466] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:30.535627 944 raft_consensus.cc:513] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.536815 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_a5955263933d4b2181efe3dcb1f49a2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:30.537129 944 leader_election.cc:290] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:30.539283 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5955263933d4b2181efe3dcb1f49a2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.540338 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet ae7c2ac546114a819a543f1667fb7844 (table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]) successfully deleted
I20250623 08:04:30.538156 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_ee8a2c2391624131a4aac14a90311aa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.546317 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee8a2c2391624131a4aac14a90311aa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.530704 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:30.547628 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet f3d32798168345edb9898751d083057c (table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]). Expected version 1 got 0
I20250623 08:04:30.547886 32058 catalog_manager.cc:5582] T 5e4678ca1c8f498c83c6bc2553882120 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:30.548660 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 0861107ab4624a05bd7a4d24b420c502 (table default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]). Expected version 1 got 0
I20250623 08:04:30.548938 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 38ecd4c243b840f4b66d8e01078dca64 (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]). Expected version 1 got 0
I20250623 08:04:30.549153 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_a6ef1bf66cd24835896dabd3af080a6a [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 1 got 0
I20250623 08:04:30.549374 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:30.549983 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:30.550369 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]). Expected version 1 got 0
I20250623 08:04:30.555132 32314 leader_election.cc:304] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:30.557354 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 65841.8us
I20250623 08:04:30.558936 952 ts_tablet_manager.cc:1918] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:30.559093 952 log.cc:1199] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/4fe30079045d437aa61bb5bfbfb96133
I20250623 08:04:30.559901 342 tablet_bootstrap.cc:492] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:30.560062 342 ts_tablet_manager.cc:1397] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.182s user 0.003s sys 0.007s
I20250623 08:04:30.560139 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:30.560732 989 raft_consensus.cc:491] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.560874 989 raft_consensus.cc:513] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.562021 610 raft_consensus.cc:2802] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.562057 989 leader_election.cc:290] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:30.562134 610 raft_consensus.cc:491] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.562250 610 raft_consensus.cc:3058] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.563014 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (table default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b]). Expected version 1 got 0
I20250623 08:04:30.563103 32314 leader_election.cc:304] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.563514 746 raft_consensus.cc:2802] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.563632 746 raft_consensus.cc:491] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.563762 746 raft_consensus.cc:3058] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.563524 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:30.564113 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5fdc9e53da264a5ca1f70e1846c906f0 (table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]). Expected version 1 got 0
I20250623 08:04:30.564700 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_7e39ecbac4a8433e8ae98b0db22c30ea [id=8b20894a91c54a3eb08d5e1b4a42d269]). Expected version 2 got 0
I20250623 08:04:30.564803 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 736eb56dae194df2af90bf33d3294a4e (DEFAULT_TABLE table=default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.565207 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736eb56dae194df2af90bf33d3294a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.567034 32488 tablet_service.cc:1468] Processing CreateTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 (DEFAULT_TABLE table=default.Table_d3f72251faee462c9c301b8aff464bb5 [id=b7671dfa27be46bb8b811b12016434e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.567356 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0b2af698b6e4ed0ae66a9bb32be7b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.570199 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]). Expected version 2 got 0
I20250623 08:04:30.571017 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 21080ab069d74457a9f7fe3aabdb3c7c (DEFAULT_TABLE table=default.Table_f1d8f7ef7cf144eaa1fd741b10eac3d6 [id=9060c1640fca461c9020722b3165d144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.571017 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 9fd478c007914d2cb09d929b50322a64 (DEFAULT_TABLE table=default.Table_9f3d2a1f5b3c41fc888725e2b7d45e0e [id=25eb4e3f3a6e42e988d2a3613ef4575f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.571270 32495 tablet_service.cc:1468] Processing CreateTablet for tablet 71fadb10e6134478b013358bba056389 (DEFAULT_TABLE table=default.Table_4a19c65595654fa98efc42626b6a9d85 [id=2979e4296458439ea5753b6ba67634ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.571470 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21080ab069d74457a9f7fe3aabdb3c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.571596 339 tablet_bootstrap.cc:492] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:30.571687 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71fadb10e6134478b013358bba056389. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.571736 339 ts_tablet_manager.cc:1397] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.343s user 0.005s sys 0.002s
I20250623 08:04:30.571944 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fd478c007914d2cb09d929b50322a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.572322 896 raft_consensus.cc:491] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.572463 896 raft_consensus.cc:513] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.573045 896 leader_election.cc:290] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.577164 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32c82141edea4068b54755a85c6d7fb8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.577744 989 raft_consensus.cc:491] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.577910 989 raft_consensus.cc:513] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.578481 989 leader_election.cc:290] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:30.579262 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 3249605158634f4eab5b9d61357c383f (DEFAULT_TABLE table=default.Table_79be3d7247284828a82f02d96faa60cc [id=c618740b8cb040bcb78c452b3d747391]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.579383 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7419cb91360f49e1b3cb7e90732558c5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:30.579548 32379 raft_consensus.cc:3058] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.579711 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3249605158634f4eab5b9d61357c383f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.586691 32495 tablet_service.cc:1468] Processing CreateTablet for tablet e602d391526d4b0eb0dbf5189afbe075 (DEFAULT_TABLE table=default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.587235 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e602d391526d4b0eb0dbf5189afbe075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.590550 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_da00c692e65345c496f4e52c106aa2e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.591301 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da00c692e65345c496f4e52c106aa2e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.596230 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_c300ef24f71d4bebaa367db14adb1b8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.596861 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c300ef24f71d4bebaa367db14adb1b8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.597512 610 raft_consensus.cc:513] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.597868 32378 raft_consensus.cc:2391] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:30.598625 342 raft_consensus.cc:357] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.598865 342 raft_consensus.cc:383] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.598860 610 leader_election.cc:290] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.598934 342 raft_consensus.cc:738] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.599146 342 consensus_queue.cc:260] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.599913 945 raft_consensus.cc:491] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.600076 945 raft_consensus.cc:513] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.600656 945 leader_election.cc:290] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.601750 746 raft_consensus.cc:513] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.601752 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 0861e45cf546487d9812caa06e2704eb (DEFAULT_TABLE table=default.Table_1671bbefe7bd4d4b86a917373288a1a6 [id=baaeae53fdcb49b7889d8379fa203c79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.601845 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 52d0a356c74f4daea9093c998b4bf73e (DEFAULT_TABLE table=default.Table_da3ed120aff049bc85b20e2200850ae3 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.602203 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0861e45cf546487d9812caa06e2704eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.602475 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52d0a356c74f4daea9093c998b4bf73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.602918 746 leader_election.cc:290] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:30.603046 32492 tablet_service.cc:1468] Processing CreateTablet for tablet e9f8aa424bbb4aa8a907259270d46656 (DEFAULT_TABLE table=default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.603215 32487 tablet_service.cc:1468] Processing CreateTablet for tablet bfd3e263464c4f2c8af9262acf211193 (DEFAULT_TABLE table=default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 [id=9f97f6deca194137a353ea221d0e3676]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.603377 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9f8aa424bbb4aa8a907259270d46656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.603690 32487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfd3e263464c4f2c8af9262acf211193. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.603917 32496 tablet_service.cc:1468] Processing CreateTablet for tablet dc0dad440b494018bc0d6cfd40940bc1 (DEFAULT_TABLE table=default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.604269 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc0dad440b494018bc0d6cfd40940bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.607051 32498 tablet_service.cc:1468] Processing CreateTablet for tablet ecb422550efd40cf82bc92fca5c25eda (DEFAULT_TABLE table=default.Table_4c1c98c205e04f4e90e5a301451870da [id=5545b396fc5746469e3a8908b0178a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.607396 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb422550efd40cf82bc92fca5c25eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.610921 32379 raft_consensus.cc:2466] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:30.611877 952 ts_tablet_manager.cc:1939] T 4fe30079045d437aa61bb5bfbfb96133 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:30.613097 952 tablet_replica.cc:331] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:30.613372 952 raft_consensus.cc:2241] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:30.613569 952 raft_consensus.cc:2270] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.614132 952 ts_tablet_manager.cc:1905] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.615159 342 ts_tablet_manager.cc:1428] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.055s user 0.003s sys 0.000s
I20250623 08:04:30.615788 342 tablet_bootstrap.cc:492] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:30.619309 981 ts_tablet_manager.cc:1918] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:30.619467 32489 tablet_service.cc:1468] Processing CreateTablet for tablet aa398354a9e44633857a75c0bbba08b3 (DEFAULT_TABLE table=default.Table_6bb7361aabf7440589a3958da502f36c [id=150ddf7cf1b141bc8586b94cf6f34aba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.602470 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 1f76ef24c69f4b17a373b242d71da8b9 (DEFAULT_TABLE table=default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.622135 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a25bdc9b7e48467fba3771365e58bcbc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.628810 32655 raft_consensus.cc:3058] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.628841 981 log.cc:1199] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/c6a97b34e386443fa19a9601621b5017
I20250623 08:04:30.628927 32378 raft_consensus.cc:2466] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:30.622210 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74842fde5152419ca35f3950e465247a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:30.623039 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ac9f38877b24d619c280a1aaa0b7b08 with delete_type TABLET_DATA_DELETED (replaced by tablet a8de2b1105564fdab386442b1de4cedb) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.629437 1001 consensus_queue.cc:1035] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.621208 339 raft_consensus.cc:357] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.629732 339 raft_consensus.cc:383] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.629837 339 raft_consensus.cc:738] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.630393 339 consensus_queue.cc:260] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.630935 339 ts_tablet_manager.cc:1428] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.059s user 0.001s sys 0.001s
I20250623 08:04:30.631217 339 tablet_bootstrap.cc:492] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:30.632745 343 tablet_bootstrap.cc:492] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.628235 32652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a25bdc9b7e48467fba3771365e58bcbc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.633577 32652 raft_consensus.cc:2466] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:30.633916 981 ts_tablet_manager.cc:1939] T c6a97b34e386443fa19a9601621b5017 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:30.635442 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f3d32798168345edb9898751d083057c (table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]). Expected version 1 got 0
I20250623 08:04:30.635859 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa398354a9e44633857a75c0bbba08b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.635833 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 38ecd4c243b840f4b66d8e01078dca64 (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]). Expected version 1 got 0
I20250623 08:04:30.636137 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:30.636453 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f02f36d6c1174a23a1f1043a431d0b6b (table default.Table_75451c4863a34537a328617c53d46cae [id=be88c6ecf20d4ab1b3757c27ce998f87]). Expected version 2 got 0
I20250623 08:04:30.636722 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:30.636760 981 tablet_replica.cc:331] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:30.636962 981 raft_consensus.cc:2241] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:30.637158 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:30.637805 1001 consensus_queue.cc:1035] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.638123 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0861107ab4624a05bd7a4d24b420c502 (table default.Table_841dcc2d3ae642a29c4cf6c8fea37e88 [id=be9754ecb40a413d9940224adbd7e2eb]). Expected version 1 got 0
I20250623 08:04:30.638378 981 raft_consensus.cc:2270] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.638955 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet c6a97b34e386443fa19a9601621b5017 (table default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca [id=c330ef9f7f0549a7abc1eeab676ec11d]) successfully deleted
I20250623 08:04:30.639933 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_a6ef1bf66cd24835896dabd3af080a6a [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 1 got 0
I20250623 08:04:30.640646 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a25bdc9b7e48467fba3771365e58bcbc" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:30.641355 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:30.642395 343 ts_tablet_manager.cc:1397] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.366s user 0.014s sys 0.020s
I20250623 08:04:30.643023 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7419cb91360f49e1b3cb7e90732558c5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:30.643138 981 ts_tablet_manager.cc:1905] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.643177 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:30.639639 339 tablet_bootstrap.cc:654] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.643855 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:30.644512 32488 tablet_service.cc:1468] Processing CreateTablet for tablet 724d662bcf754c74a8f3dff91ee258ae (DEFAULT_TABLE table=default.Table_9fc75fd22c6a467782fe3b4c3b123602 [id=1a49c0c47b6647ef8fc4c7c20ad9bc1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.644563 32519 raft_consensus.cc:2466] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:30.644932 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 18ea63263b9b466e922169eecafab747 (DEFAULT_TABLE table=default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.647440 32652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a25bdc9b7e48467fba3771365e58bcbc" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.647616 32652 raft_consensus.cc:2466] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:30.648316 32654 raft_consensus.cc:3058] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.649492 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ac9f38877b24d619c280a1aaa0b7b08 with delete_type TABLET_DATA_DELETED (replaced by tablet a8de2b1105564fdab386442b1de4cedb) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.649173 657 consensus_queue.cc:1035] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250623 08:04:30.650128 342 tablet_bootstrap.cc:654] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.650439 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 0e9bf7cba2244703b7402cb4be1a005e (DEFAULT_TABLE table=default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.651130 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]). Expected version 1 got 0
I20250623 08:04:30.650715 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32c82141edea4068b54755a85c6d7fb8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:30.651549 32651 raft_consensus.cc:3058] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.652098 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]). Expected version 2 got 0
I20250623 08:04:30.648762 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74842fde5152419ca35f3950e465247a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:30.652660 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d02f74c24a174b18b08a870a4e47fefb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:30.653178 32650 raft_consensus.cc:3058] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.653522 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (table default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b]). Expected version 1 got 0
I20250623 08:04:30.655120 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:30.655441 32653 raft_consensus.cc:3058] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.655690 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5fdc9e53da264a5ca1f70e1846c906f0 (table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]). Expected version 1 got 0
I20250623 08:04:30.656193 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_7e39ecbac4a8433e8ae98b0db22c30ea [id=8b20894a91c54a3eb08d5e1b4a42d269]). Expected version 2 got 0
I20250623 08:04:30.656471 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f76ef24c69f4b17a373b242d71da8b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.656737 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18ea63263b9b466e922169eecafab747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.657295 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32c82141edea4068b54755a85c6d7fb8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:30.657508 32517 raft_consensus.cc:3058] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.658550 989 consensus_queue.cc:1035] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250623 08:04:30.657966 32052 catalog_manager.cc:5582] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:30.660368 32052 catalog_manager.cc:5582] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:30.660921 944 raft_consensus.cc:491] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.662029 944 raft_consensus.cc:513] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.662720 944 leader_election.cc:290] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:30.665210 32057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdc9d47f2232403fa71e14a6fe6f33c3 consensus state has no leader
I20250623 08:04:30.674790 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 5f1fb8231e8e45208ac4f412134d8696 (DEFAULT_TABLE table=default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.675074 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9bf7cba2244703b7402cb4be1a005e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.677991 32453 leader_election.cc:304] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.679791 32650 raft_consensus.cc:2466] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:30.680102 32654 raft_consensus.cc:2466] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:30.680291 1001 consensus_queue.cc:1035] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.680701 343 raft_consensus.cc:357] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.681012 32314 leader_election.cc:304] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.681489 343 raft_consensus.cc:383] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 08:04:30.681537 32054 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 358cb6ac24414bdfbc32155e732f9ac8 consensus state has no leader
I20250623 08:04:30.681589 1001 raft_consensus.cc:2802] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.681694 1001 raft_consensus.cc:491] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.681771 1001 raft_consensus.cc:3058] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.681968 32057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c64abc5f2e1240bba6400c28a19631a6" } new_table_name: "default.Table_32db9f68dd974fb8aff372a8eeccce3d" modify_external_catalogs: true
W20250623 08:04:30.684590 32053 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7fcf583064f4538913d9157e93068dd consensus state has no leader
I20250623 08:04:30.684775 1001 raft_consensus.cc:513] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.685295 1001 leader_election.cc:290] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.685523 32053 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_244bcdc8d69b4f61a337dac403ffe6cb" } modify_external_catalogs: true
I20250623 08:04:30.685729 32053 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_244bcdc8d69b4f61a337dac403ffe6cb" } modify_external_catalogs: true
I20250623 08:04:30.686596 657 raft_consensus.cc:2802] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:30.687260 32453 leader_election.cc:304] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.688372 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 724d662bcf754c74a8f3dff91ee258ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:30.688757 32453 leader_election.cc:343] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.688733 874 consensus_queue.cc:1035] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250623 08:04:30.676501 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d02f74c24a174b18b08a870a4e47fefb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:30.680902 32651 raft_consensus.cc:2466] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:30.691771 1001 consensus_queue.cc:1035] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.692348 32314 leader_election.cc:304] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.692412 657 raft_consensus.cc:695] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:30.692873 1001 raft_consensus.cc:2802] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:30.692974 657 consensus_queue.cc:237] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.693109 1001 raft_consensus.cc:695] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:30.693331 1001 consensus_queue.cc:237] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.693843 699 raft_consensus.cc:2802] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.695363 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7fcf583064f4538913d9157e93068dd" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.696105 32516 raft_consensus.cc:3058] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.696655 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b81263a9ee9d4da1ac721c3b2bd21248" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.696835 32380 raft_consensus.cc:2466] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:30.697167 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7fcf583064f4538913d9157e93068dd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.697329 32378 raft_consensus.cc:2466] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:30.698282 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 4fe30079045d437aa61bb5bfbfb96133 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]) successfully deleted
W20250623 08:04:30.700861 32057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42c861e29e1e4e549021dae7255a856c has no consensus state
I20250623 08:04:30.701257 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_b7a92f8a00ba4931827eaee759bc8f91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.701866 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7a92f8a00ba4931827eaee759bc8f91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.702497 979 ts_tablet_manager.cc:1918] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:30.707669 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31a67b70cbc64bb3bfd5a34865109a01" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:30.708058 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32c82141edea4068b54755a85c6d7fb8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.708271 32380 raft_consensus.cc:2391] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:30.709086 32516 raft_consensus.cc:2466] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:30.709118 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0545daa6fba846adbc573a63d8dbcd66" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.709276 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417a050eada4403ebbddff255fb6fb5a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.705556 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f1fb8231e8e45208ac4f412134d8696. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.710124 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "358cb6ac24414bdfbc32155e732f9ac8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.711046 945 raft_consensus.cc:491] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.713747 32453 leader_election.cc:304] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.714354 699 raft_consensus.cc:491] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.714490 699 raft_consensus.cc:3058] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.714799 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_fbb1fc5910454112b795f72c26f1af5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.715474 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbb1fc5910454112b795f72c26f1af5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.719836 32653 raft_consensus.cc:2466] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:30.723428 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d3f72251faee462c9c301b8aff464bb5" } modify_external_catalogs: true
I20250623 08:04:30.723626 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d3f72251faee462c9c301b8aff464bb5" } modify_external_catalogs: true
I20250623 08:04:30.724773 32056 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 0311ce67989f49e8866234a88e5b4d09
I20250623 08:04:30.725471 32056 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fd49e04fcac14f0194cc9fed910601fc" } modify_external_catalogs: true
I20250623 08:04:30.725663 32056 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fd49e04fcac14f0194cc9fed910601fc" } modify_external_catalogs: true
I20250623 08:04:30.726995 32051 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 965f1f0a107b4653b6aac01e0d8e259a
I20250623 08:04:30.727479 32051 master_service.cc:948] ReplaceTablet: received request to replace tablet 7da84969a6464b5caeaf47af0cbaaa4d from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:30.728322 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adc3cf4355c24b7395fcb22af8520346" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:30.728540 32655 raft_consensus.cc:2466] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:30.728986 32650 raft_consensus.cc:2466] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:30.729336 343 raft_consensus.cc:738] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.729637 343 consensus_queue.cc:260] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.730273 343 ts_tablet_manager.cc:1428] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.088s user 0.000s sys 0.002s
I20250623 08:04:30.730763 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7fcf583064f4538913d9157e93068dd" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.731016 32378 raft_consensus.cc:2466] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:30.730633 343 tablet_bootstrap.cc:492] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.731801 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet d0b2af698b6e4ed0ae66a9bb32be7b31
I20250623 08:04:30.732502 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_c76a280c52214058b9b4b2691090cc1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:30.733028 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adc3cf4355c24b7395fcb22af8520346" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:30.733093 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c76a280c52214058b9b4b2691090cc1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.734197 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a25bdc9b7e48467fba3771365e58bcbc" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:30.734406 32653 raft_consensus.cc:3058] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.741271 32653 raft_consensus.cc:2466] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:30.742784 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_841dcc2d3ae642a29c4cf6c8fea37e88" } new_table_name: "default.Table_977fe8fc665f4267b394065e22f67eea" modify_external_catalogs: true
I20250623 08:04:30.745529 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 0311ce67989f49e8866234a88e5b4d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.745674 32057 master_service.cc:948] ReplaceTablet: received request to replace tablet d6ed4cb252b1452087f8ee53ec78d985 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:30.746917 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.746834 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.747493 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.748090 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.748342 32314 leader_election.cc:304] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.752890 981 ts_tablet_manager.cc:1918] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:30.753299 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_fcc07b378bf64122ae65bc17f130b2f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.754144 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcc07b378bf64122ae65bc17f130b2f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.759872 343 tablet_bootstrap.cc:654] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.760180 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a25bdc9b7e48467fba3771365e58bcbc" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:30.763945 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_04e85a0dbf974a05a94b9a9713fcb3f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.764688 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04e85a0dbf974a05a94b9a9713fcb3f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.753801 1000 raft_consensus.cc:2802] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:30.766131 1000 raft_consensus.cc:695] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:30.766561 1000 consensus_queue.cc:237] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.755571 610 consensus_queue.cc:1035] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.755944 32314 leader_election.cc:304] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.771029 746 raft_consensus.cc:2802] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:30.771229 746 raft_consensus.cc:695] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:30.771591 746 consensus_queue.cc:237] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:30.771622 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 58906.3us
I20250623 08:04:30.772665 32589 leader_election.cc:304] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.772984 32589 leader_election.cc:304] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:30.773270 32589 leader_election.cc:343] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:30.773571 32589 leader_election.cc:343] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:30.773809 32589 leader_election.cc:343] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.774224 990 raft_consensus.cc:2802] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.774327 990 raft_consensus.cc:491] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.774477 990 raft_consensus.cc:3058] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.777496 990 raft_consensus.cc:513] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.780009 1011 raft_consensus.cc:2802] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.780115 1011 raft_consensus.cc:491] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.780225 1011 raft_consensus.cc:3058] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.778086 990 leader_election.cc:290] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.783401 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 0311ce67989f49e8866234a88e5b4d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.786727 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_f5479d2204fd477ba893d16a0f77c4fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.787645 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5479d2204fd477ba893d16a0f77c4fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.794216 343 tablet_bootstrap.cc:492] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.800781 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_3188d5c072e04adea5dae4c5ec7a71c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.801716 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3188d5c072e04adea5dae4c5ec7a71c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:30.802141 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 61137.3us
I20250623 08:04:30.803751 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b81263a9ee9d4da1ac721c3b2bd21248" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:30.804416 32379 raft_consensus.cc:3058] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.804839 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_bf77bbe8745846d3b962bd5b7fed73ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.805470 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf77bbe8745846d3b962bd5b7fed73ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.807291 343 ts_tablet_manager.cc:1397] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.003s
I20250623 08:04:30.807825 944 raft_consensus.cc:491] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.811046 944 raft_consensus.cc:513] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.811754 944 leader_election.cc:290] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.807950 945 raft_consensus.cc:513] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.812318 945 leader_election.cc:290] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.808145 979 log.cc:1199] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/c0f87fd62f1545089d00757f8214593a
I20250623 08:04:30.809679 710 raft_consensus.cc:491] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:30.813854 710 raft_consensus.cc:513] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.813925 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44fd3c67185484e93450682364dc8a8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.814509 710 leader_election.cc:290] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.809830 819 consensus_queue.cc:1035] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.098s
I20250623 08:04:30.815397 979 ts_tablet_manager.cc:1939] T c0f87fd62f1545089d00757f8214593a P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:30.818161 32589 leader_election.cc:343] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.818888 343 raft_consensus.cc:357] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.819082 343 raft_consensus.cc:383] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.819157 343 raft_consensus.cc:738] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.820717 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet c0f87fd62f1545089d00757f8214593a (table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]) successfully deleted
I20250623 08:04:30.821388 1011 raft_consensus.cc:513] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.821985 1011 leader_election.cc:290] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:30.826893 343 consensus_queue.cc:260] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.827423 343 ts_tablet_manager.cc:1428] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250623 08:04:30.828732 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7fcf583064f4538913d9157e93068dd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:30.828986 32381 raft_consensus.cc:3058] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.829916 343 tablet_bootstrap.cc:492] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:30.830824 979 tablet_replica.cc:331] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:30.831662 979 raft_consensus.cc:2241] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:30.831846 979 raft_consensus.cc:2270] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.832387 979 ts_tablet_manager.cc:1905] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.832592 32381 raft_consensus.cc:2466] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:30.833087 32589 leader_election.cc:304] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.833364 1011 raft_consensus.cc:2802] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:30.833626 1011 raft_consensus.cc:695] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:30.834239 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:30.810714 32379 raft_consensus.cc:2466] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
W20250623 08:04:30.834709 32589 leader_election.cc:343] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.835144 1011 consensus_queue.cc:237] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.836119 32056 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f3c850021e914530bb106ab7166be7c9
I20250623 08:04:30.836891 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f4a0b9402a264800a1ea39707c10da0c
I20250623 08:04:30.838840 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet d6ed4cb252b1452087f8ee53ec78d985
I20250623 08:04:30.839417 32057 catalog_manager.cc:6652] ReplaceTablet: tablet d6ed4cb252b1452087f8ee53ec78d985 deleted and replaced by tablet 306c4021b5d94a4996de07c222a8bfd8
I20250623 08:04:30.840070 848 consensus_queue.cc:1035] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20250623 08:04:30.840338 32028 master_runner.cc:186] Error getting master registration for 127.31.69.61:33313: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111),
I20250623 08:04:30.840492 32028 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250623 08:04:30.845167 699 raft_consensus.cc:513] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.845299 32589 leader_election.cc:304] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:30.845624 819 raft_consensus.cc:2802] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:30.845775 699 leader_election.cc:290] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:30.845793 819 raft_consensus.cc:695] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:30.846092 819 consensus_queue.cc:237] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
W20250623 08:04:30.848187 32058 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6f9e0d1b85a94ae9ae86e53688a55db5 consensus state has no leader
W20250623 08:04:30.849620 32058 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 283bebc7063d437ba3e541d7dee3c0b5 consensus state has no leader
W20250623 08:04:30.849926 32058 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet d02f74c24a174b18b08a870a4e47fefb consensus state has no leader
I20250623 08:04:30.849952 32516 raft_consensus.cc:2391] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:30.851477 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_6a8ea07f0b174864b81addf035ed1f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:30.852416 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a8ea07f0b174864b81addf035ed1f34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.853677 32517 raft_consensus.cc:2466] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:30.845041 989 consensus_queue.cc:1035] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20250623 08:04:30.855645 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 2060a643d0da47efa868f52bdb331b1f with delete_type TABLET_DATA_DELETED (replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.856084 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.856518 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 279624b8bb834b95a4f08045968ebfb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.857100 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 19536c16ff32460e9db571e4ce411bab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.857545 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e47b10f46b8402e94354ebca573c194 with delete_type TABLET_DATA_DELETED (replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.857900 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 417a050eada4403ebbddff255fb6fb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.858063 874 raft_consensus.cc:2762] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250623 08:04:30.858285 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 175ee90133994154b7e4b58b0770db3b with delete_type TABLET_DATA_DELETED (replaced by tablet e602d391526d4b0eb0dbf5189afbe075) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.858698 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 67c44697ecab4dd9b43f544a602897ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.859161 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet a414144276f6484ea4690204752e2142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.859540 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d with delete_type TABLET_DATA_DELETED (replaced by tablet 136e482995a24c5daec55a701f3091fa) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.859937 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 with delete_type TABLET_DATA_DELETED (replaced by tablet 205da880685b49b3b8669f73dc2c563b) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.860204 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 with delete_type TABLET_DATA_DELETED (replaced by tablet 306c4021b5d94a4996de07c222a8bfd8) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.860486 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a0b9402a264800a1ea39707c10da0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.860741 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c850021e914530bb106ab7166be7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:30.861047 32058 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet f3d32798168345edb9898751d083057c (table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f3d32798168345edb9898751d083057c consensus state has no leader
W20250623 08:04:30.862042 32058 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 38ecd4c243b840f4b66d8e01078dca64 (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 38ecd4c243b840f4b66d8e01078dca64 consensus state has no leader
W20250623 08:04:30.862439 32058 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_a6ef1bf66cd24835896dabd3af080a6a [id=33768fa88bfc4cfba1359030af5e77c9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet c63029399994480ba7ccabcadc1ee5e0 consensus state has no leader
W20250623 08:04:30.863023 32058 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet c0773fc60d4a463596f536f15d58bf57 consensus state has no leader
W20250623 08:04:30.863564 32058 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 consensus state has no leader
I20250623 08:04:30.865404 32624 tablet_service.cc:1515] Processing DeleteTablet for tablet b980c746a3c042bfae90aa076acbb1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.865942 32624 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ecf4af9e476495683bd63cac20d4672 with delete_type TABLET_DATA_DELETED (replaced by tablet 52d0a356c74f4daea9093c998b4bf73e) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.866142 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 228375592aee48a1aa21e1d16962b663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.866885 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet 32c82141edea4068b54755a85c6d7fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 162308da898d4dde8ab5b1ab01d0f393) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.867172 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.867239 32053 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_7e39ecbac4a8433e8ae98b0db22c30ea" } new_table_name: "default.Table_3af33890e8ff40c881af74580254633b" modify_external_catalogs: true
I20250623 08:04:30.867523 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.871920 952 ts_tablet_manager.cc:1918] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:30.874186 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet 726d80e05a344f27b5a289723cb5aed7 with delete_type TABLET_DATA_DELETED (replaced by tablet 432d1642c0e44119b5bc05858463a136) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.874472 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f87fd62f1545089d00757f8214593a with delete_type TABLET_DATA_DELETED (replaced by tablet 1f76ef24c69f4b17a373b242d71da8b9) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.874739 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.875352 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.875592 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet 966a159e212040109e2842c15b69fb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.875828 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dc696236a15420e9666cca708068fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.876098 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet 162308da898d4dde8ab5b1ab01d0f393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:30.878173 342 tablet_bootstrap.cc:492] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:30.878340 342 ts_tablet_manager.cc:1397] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.263s user 0.016s sys 0.020s
I20250623 08:04:30.879552 343 tablet_bootstrap.cc:654] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.879284 952 log.cc:1199] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/c0f87fd62f1545089d00757f8214593a
I20250623 08:04:30.881867 342 raft_consensus.cc:357] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.882225 342 raft_consensus.cc:383] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.883873 342 raft_consensus.cc:738] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.884125 342 consensus_queue.cc:260] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.887754 342 ts_tablet_manager.cc:1428] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:30.888902 952 ts_tablet_manager.cc:1939] T c0f87fd62f1545089d00757f8214593a P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:30.893342 32059 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a1c82e4d2db64026a04c5c84bc1e09ec consensus state has no leader
I20250623 08:04:30.893965 342 tablet_bootstrap.cc:492] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:30.895238 32059 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 31ecc74317d449e5b557b07a4fd14895 consensus state has no leader
I20250623 08:04:30.895781 342 tablet_bootstrap.cc:654] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:30.896128 32059 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet a25bdc9b7e48467fba3771365e58bcbc consensus state has no leader
W20250623 08:04:30.898718 32059 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:39501 (request call id 165) took 865 ms (client timeout 1000 ms). Trace:
I20250623 08:04:30.900475 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_4acf3aff222e477a885bc37d00623ddf" } new_table_name: "default.Table_d496fd69b6f140b88b5abc86c8839ee2" modify_external_catalogs: true
I20250623 08:04:30.902164 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_9f3d2a1f5b3c41fc888725e2b7d45e0e" } new_table_name: "default.Table_087d39ad116a4ac2a1cf1050403ef202" modify_external_catalogs: true
I20250623 08:04:30.904443 32056 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_38934e8ef2e34a6a94c27c2058a1803f" } new_table_name: "default.Table_dc973f39a76c44a1972c2b9a6c9328b8" modify_external_catalogs: true
I20250623 08:04:30.908788 979 ts_tablet_manager.cc:1918] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:30.908921 979 log.cc:1199] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/4ac9f38877b24d619c280a1aaa0b7b08
I20250623 08:04:30.909320 979 ts_tablet_manager.cc:1939] T 4ac9f38877b24d619c280a1aaa0b7b08 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:30.910707 952 tablet_replica.cc:331] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:30.910985 952 raft_consensus.cc:2241] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:30.911190 952 raft_consensus.cc:2270] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.911911 952 ts_tablet_manager.cc:1905] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.915256 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67c44697ecab4dd9b43f544a602897ca" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:30.918061 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 113556us
I20250623 08:04:30.918656 657 consensus_queue.cc:1035] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.919211 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 4ac9f38877b24d619c280a1aaa0b7b08 (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]) successfully deleted
I20250623 08:04:30.919549 657 consensus_queue.cc:1035] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:30.920068 32623 tablet_service.cc:1468] Processing CreateTablet for tablet 87648b11b10b45fc9cc8c998bbceae92 (DEFAULT_TABLE table=default.Table_2fceb5254f914cd9a952ae0a805150c5 [id=ad88a05d1bbb448e94daa1471a373cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.920280 952 ts_tablet_manager.cc:1918] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:30.920529 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87648b11b10b45fc9cc8c998bbceae92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.920823 32453 leader_election.cc:304] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:30.921155 657 raft_consensus.cc:2802] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:30.921782 657 raft_consensus.cc:695] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:30.921865 342 tablet_bootstrap.cc:492] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:30.922264 657 consensus_queue.cc:237] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:30.923870 32453 leader_election.cc:304] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:30.924186 342 ts_tablet_manager.cc:1397] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.009s
I20250623 08:04:30.924976 342 raft_consensus.cc:357] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.925182 342 raft_consensus.cc:383] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.925267 342 raft_consensus.cc:738] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.925519 342 consensus_queue.cc:260] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.925822 874 raft_consensus.cc:2802] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.925920 874 raft_consensus.cc:491] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.926025 874 raft_consensus.cc:3058] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.926076 342 ts_tablet_manager.cc:1428] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:30.926373 342 tablet_bootstrap.cc:492] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:30.926817 952 log.cc:1199] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/4ac9f38877b24d619c280a1aaa0b7b08
I20250623 08:04:30.927265 952 ts_tablet_manager.cc:1939] T 4ac9f38877b24d619c280a1aaa0b7b08 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:30.927699 32453 leader_election.cc:304] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:30.929657 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a25bdc9b7e48467fba3771365e58bcbc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:30.929888 32655 raft_consensus.cc:2391] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:30.929893 874 raft_consensus.cc:513] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.930440 874 leader_election.cc:290] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:30.930732 874 raft_consensus.cc:2802] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:30.931442 874 raft_consensus.cc:491] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:30.931620 874 raft_consensus.cc:3058] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:30.932962 32453 leader_election.cc:343] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:30.935145 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.935966 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.938215 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c850021e914530bb106ab7166be7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.938457 952 tablet_replica.cc:331] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:30.938676 952 raft_consensus.cc:2241] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:30.938899 952 pending_rounds.cc:70] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:30.938958 952 pending_rounds.cc:77] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727400669351936 op_type: NO_OP noop_request { }
I20250623 08:04:30.939090 952 raft_consensus.cc:2887] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:30.939185 952 raft_consensus.cc:2270] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.939736 952 ts_tablet_manager.cc:1905] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.940644 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 0311ce67989f49e8866234a88e5b4d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:30.941242 32058 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f consensus state has no leader
I20250623 08:04:30.943096 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a0b9402a264800a1ea39707c10da0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.944430 342 tablet_bootstrap.cc:654] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:30.946187 32653 raft_consensus.cc:3058] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:30.953794 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 with delete_type TABLET_DATA_DELETED (replaced by tablet 306c4021b5d94a4996de07c222a8bfd8) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.955258 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet c6a97b34e386443fa19a9601621b5017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.955513 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.955596 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ecf4af9e476495683bd63cac20d4672 with delete_type TABLET_DATA_DELETED (replaced by tablet 52d0a356c74f4daea9093c998b4bf73e) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.956282 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.957146 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 2060a643d0da47efa868f52bdb331b1f with delete_type TABLET_DATA_DELETED (replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.958431 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet d4221d7500d1458db75926d7b274a654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.958774 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 36c8714cfbd84dd38cf5c0f305ef6843 (DEFAULT_TABLE table=default.Table_2092a828aea549a38d1bd42248c8804c [id=cce6d993b26f4f9ca197d807107e00cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.958870 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 279624b8bb834b95a4f08045968ebfb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.958933 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.959224 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c8714cfbd84dd38cf5c0f305ef6843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.959635 32623 tablet_service.cc:1468] Processing CreateTablet for tablet a38eecf4a94d4536b98d4f6e037a0094 (DEFAULT_TABLE table=default.Table_e216bbb5ac2a4720b36183e30bd6ed91 [id=57e548bc818e4968b432cc3af298e675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.960067 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38eecf4a94d4536b98d4f6e037a0094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.960618 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 32c82141edea4068b54755a85c6d7fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 162308da898d4dde8ab5b1ab01d0f393) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.961035 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 726d80e05a344f27b5a289723cb5aed7 with delete_type TABLET_DATA_DELETED (replaced by tablet 432d1642c0e44119b5bc05858463a136) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.961416 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet b980c746a3c042bfae90aa076acbb1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.961664 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.961787 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet a414144276f6484ea4690204752e2142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.962174 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 67c44697ecab4dd9b43f544a602897ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.962910 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dc696236a15420e9666cca708068fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.963461 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.964150 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe30079045d437aa61bb5bfbfb96133 with delete_type TABLET_DATA_DELETED (replaced by tablet 18ea63263b9b466e922169eecafab747) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.964123 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c850021e914530bb106ab7166be7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.964536 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a0b9402a264800a1ea39707c10da0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.964818 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 162308da898d4dde8ab5b1ab01d0f393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.964927 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 with delete_type TABLET_DATA_DELETED (replaced by tablet 306c4021b5d94a4996de07c222a8bfd8) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:30.965436 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 205da880685b49b3b8669f73dc2c563b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.966152 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 with delete_type TABLET_DATA_DELETED (replaced by tablet 31a67b70cbc64bb3bfd5a34865109a01) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.967259 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f with delete_type TABLET_DATA_DELETED (replaced by tablet b1aa596950da49a898e3ed9bbad8c45e) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.968602 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ea81642050141a9be84f151a5785abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.969044 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.969481 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_e8b3bfa63ee7407ead7f4a0f8204d269"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:30.969723 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 776b412b3f564a7581c6a5e1c84e8499 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:30.970227 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8b3bfa63ee7407ead7f4a0f8204d269 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:30.970338 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.971009 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet cde7df7d39a54114a4dcb28cfd1eb429 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f1fb8231e8e45208ac4f412134d8696) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.972225 32055 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a5955263933d4b2181efe3dcb1f49a2d" } modify_external_catalogs: true
I20250623 08:04:30.972625 32055 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a5955263933d4b2181efe3dcb1f49a2d" } modify_external_catalogs: true
I20250623 08:04:30.974197 979 tablet_replica.cc:331] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:30.974192 952 ts_tablet_manager.cc:1918] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:30.974345 952 log.cc:1199] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f3c850021e914530bb106ab7166be7c9
I20250623 08:04:30.974452 979 raft_consensus.cc:2241] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:30.974623 979 raft_consensus.cc:2270] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:30.975311 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet ae7c2ac546114a819a543f1667fb7844 with delete_type TABLET_DATA_DELETED (replaced by tablet d98654c29b9c4e75af6a213cafd4c2c9) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.975374 979 ts_tablet_manager.cc:1905] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:30.976195 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f9e0d1b85a94ae9ae86e53688a55db5) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.976966 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d with delete_type TABLET_DATA_DELETED (replaced by tablet 136e482995a24c5daec55a701f3091fa) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.978051 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 966a159e212040109e2842c15b69fb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.979080 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet f0117cbb09dd4067836266c9d6a32adf with delete_type TABLET_DATA_DELETED (replaced by tablet 2ecf4af9e476495683bd63cac20d4672) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.979681 343 tablet_bootstrap.cc:492] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:30.979864 343 ts_tablet_manager.cc:1397] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.152s user 0.000s sys 0.008s
I20250623 08:04:30.980315 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 175ee90133994154b7e4b58b0770db3b with delete_type TABLET_DATA_DELETED (replaced by tablet e602d391526d4b0eb0dbf5189afbe075) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.980978 343 raft_consensus.cc:357] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.981205 343 raft_consensus.cc:383] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:30.981588 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 with delete_type TABLET_DATA_DELETED (replaced by tablet 205da880685b49b3b8669f73dc2c563b) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.981307 343 raft_consensus.cc:738] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:30.981841 343 consensus_queue.cc:260] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.982724 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 417a050eada4403ebbddff255fb6fb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.982722 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 42c861e29e1e4e549021dae7255a856c (DEFAULT_TABLE table=default.Table_32db9f68dd974fb8aff372a8eeccce3d [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.983139 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c861e29e1e4e549021dae7255a856c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.983523 32624 tablet_service.cc:1468] Processing CreateTablet for tablet f4a96ae9190640dea11bd11a9f0e4372 (DEFAULT_TABLE table=default.Table_0666030ad7ec4ef89335cdbaf0cca443 [id=bf6ca5aa8d5044868244a9b37cfa9519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.983726 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 19536c16ff32460e9db571e4ce411bab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.983884 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a96ae9190640dea11bd11a9f0e4372. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.984578 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e47b10f46b8402e94354ebca573c194 with delete_type TABLET_DATA_DELETED (replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:30.985426 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 87648b11b10b45fc9cc8c998bbceae92 (DEFAULT_TABLE table=default.Table_2fceb5254f914cd9a952ae0a805150c5 [id=ad88a05d1bbb448e94daa1471a373cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.985831 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87648b11b10b45fc9cc8c998bbceae92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.986027 874 raft_consensus.cc:513] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:30.987013 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 2b405f87bb1e4d0686a1f627835d901f (DEFAULT_TABLE table=default.Table_f40b881c21294c10b35962b75a30524c [id=bdca6a821a6b48cbb3366d8912abeeda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.987387 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b405f87bb1e4d0686a1f627835d901f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.990631 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 7874a1d5cb9a464294ca32b2767efbf3 (DEFAULT_TABLE table=default.Table_8ec723069381477ca64f5aaad0df76fb [id=2a5a01b05a554ac1ad6abd21ad00d734]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.991003 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7874a1d5cb9a464294ca32b2767efbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.993958 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b81263a9ee9d4da1ac721c3b2bd21248" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:30.994206 32517 raft_consensus.cc:2466] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:30.994181 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 7874a1d5cb9a464294ca32b2767efbf3 (DEFAULT_TABLE table=default.Table_8ec723069381477ca64f5aaad0df76fb [id=2a5a01b05a554ac1ad6abd21ad00d734]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:30.898910 32059 rpcz_store.cc:269] 0623 08:04:30.033311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.148384 (+115073us) service_pool.cc:225] Handling call
0623 08:04:30.191006 (+ 42622us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f3b58 00007f19c7640176 00007f19c763fea5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:30.476066 (+285060us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:30.476072 (+ 6us) write_op.cc:454] Released schema lock
0623 08:04:30.898674 (+422602us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:30.465908 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:30.466032 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:30.466041 (+ 9us) write_op.cc:435] Acquired schema lock
0623 08:04:30.466044 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:30.466357 (+ 313us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:30.466373 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:30.466380 (+ 7us) tablet.cc:683] Acquiring locks for 10 operations
0623 08:04:30.466589 (+ 209us) tablet.cc:699] Row locks acquired
0623 08:04:30.466592 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:30.466635 (+ 43us) write_op.cc:270] Start()
0623 08:04:30.466673 (+ 38us) write_op.cc:276] Timestamp: P: 1750665870466585 usec, L: 0
0623 08:04:30.466680 (+ 7us) op_driver.cc:348] REPLICATION: starting
0623 08:04:30.466996 (+ 316us) log.cc:844] Serialized 3894 byte log entry
0623 08:04:30.473167 (+ 6171us) op_driver.cc:464] REPLICATION: finished
0623 08:04:30.473322 (+ 155us) write_op.cc:301] APPLY: starting
0623 08:04:30.473339 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:30.473457 (+ 118us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:30.473513 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:30.473717 (+ 204us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:30.473758 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0623 08:04:30.473767 (+ 9us) write_op.cc:312] APPLY: finished
0623 08:04:30.473896 (+ 129us) log.cc:844] Serialized 77 byte log entry
0623 08:04:30.473960 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:30.473993 (+ 33us) write_op.cc:454] Released schema lock
0623 08:04:30.474000 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":43566,"reported_tablets":99,"spinlock_wait_cycles":81077504,"child_traces":[["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":742,"apply.run_wall_time_us":742,"num_ops":10,"prepare.queue_time_us":10657,"prepare.run_cpu_time_us":1226,"prepare.run_wall_time_us":1226,"raft.queue_time_us":558,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":358,"replication_time_us":6448,"spinlock_wait_cycles":140288}]]}
I20250623 08:04:30.993958 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 2b405f87bb1e4d0686a1f627835d901f (DEFAULT_TABLE table=default.Table_f40b881c21294c10b35962b75a30524c [id=bdca6a821a6b48cbb3366d8912abeeda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:30.999574 32059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_4ee0e165259b4e2c88626b2d613149ca" } modify_external_catalogs: true
I20250623 08:04:30.999715 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b405f87bb1e4d0686a1f627835d901f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:30.999909 32059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_4ee0e165259b4e2c88626b2d613149ca" } modify_external_catalogs: true
I20250623 08:04:30.999033 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 42c861e29e1e4e549021dae7255a856c (DEFAULT_TABLE table=default.Table_32db9f68dd974fb8aff372a8eeccce3d [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:31.004997 32058 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:39129 (request call id 167) took 993 ms (client timeout 1000 ms). Trace:
W20250623 08:04:31.005208 32058 rpcz_store.cc:269] 0623 08:04:30.011581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.146490 (+134909us) service_pool.cc:225] Handling call
0623 08:04:30.198964 (+ 52474us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f3b58 00007f19c7640176 00007f19c763fea5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:31.004929 (+805965us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:30.584105 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:30.584224 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:30.584232 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:30.584235 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:30.584353 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:30.584367 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:30.584373 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:30.584443 (+ 70us) tablet.cc:699] Row locks acquired
0623 08:04:30.584445 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:30.584480 (+ 35us) write_op.cc:270] Start()
0623 08:04:30.584518 (+ 38us) write_op.cc:276] Timestamp: P: 1750665870584420 usec, L: 0
0623 08:04:30.584522 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:30.584723 (+ 201us) log.cc:844] Serialized 527 byte log entry
0623 08:04:30.600393 (+ 15670us) op_driver.cc:464] REPLICATION: finished
0623 08:04:30.601017 (+ 624us) write_op.cc:301] APPLY: starting
0623 08:04:30.601038 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:30.601092 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:30.622941 (+ 21849us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:30.623030 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:30.623067 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:30.623076 (+ 9us) write_op.cc:312] APPLY: finished
0623 08:04:30.623208 (+ 132us) log.cc:844] Serialized 23 byte log entry
0623 08:04:30.623322 (+ 114us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:30.623351 (+ 29us) write_op.cc:454] Released schema lock
0623 08:04:30.623359 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:30.623445 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:30.623452 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":104,"spinlock_wait_cycles":92197120,"child_traces":[["op",{"apply.queue_time_us":539,"apply.run_cpu_time_us":674,"apply.run_wall_time_us":22422,"num_ops":1,"prepare.queue_time_us":381,"prepare.run_cpu_time_us":731,"prepare.run_wall_time_us":730,"raft.queue_time_us":36,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":15843}]]}
I20250623 08:04:31.007344 32058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d895ba420734421e8b8f074848f4e389" } modify_external_catalogs: true
I20250623 08:04:31.007655 32058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d895ba420734421e8b8f074848f4e389" } modify_external_catalogs: true
I20250623 08:04:31.009512 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ac9f38877b24d619c280a1aaa0b7b08 with delete_type TABLET_DATA_DELETED (replaced by tablet a8de2b1105564fdab386442b1de4cedb) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.009948 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 205da880685b49b3b8669f73dc2c563b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.010416 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 with delete_type TABLET_DATA_DELETED (replaced by tablet 31a67b70cbc64bb3bfd5a34865109a01) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.010845 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f with delete_type TABLET_DATA_DELETED (replaced by tablet b1aa596950da49a898e3ed9bbad8c45e) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.011205 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f9e0d1b85a94ae9ae86e53688a55db5) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.011592 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 04eda92e14144733ab71a577dd3bd2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.011925 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet f0117cbb09dd4067836266c9d6a32adf with delete_type TABLET_DATA_DELETED (replaced by tablet 2ecf4af9e476495683bd63cac20d4672) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.012274 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d with delete_type TABLET_DATA_DELETED (replaced by tablet c7fcf583064f4538913d9157e93068dd) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.012653 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet d4221d7500d1458db75926d7b274a654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.014076 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet c6a97b34e386443fa19a9601621b5017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.014602 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet ae7c2ac546114a819a543f1667fb7844 with delete_type TABLET_DATA_DELETED (replaced by tablet d98654c29b9c4e75af6a213cafd4c2c9) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.014981 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 776b412b3f564a7581c6a5e1c84e8499 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:31.015249 32056 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdc9d47f2232403fa71e14a6fe6f33c3 consensus state has no leader
I20250623 08:04:31.015280 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.015600 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet cde7df7d39a54114a4dcb28cfd1eb429 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f1fb8231e8e45208ac4f412134d8696) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.015882 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ea81642050141a9be84f151a5785abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.016219 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.016561 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.016804 32624 tablet_service.cc:1515] Processing DeleteTablet for tablet d0ee5a4fe03848229524cc0105226d8c with delete_type TABLET_DATA_DELETED (replaced by tablet 38ecd4c243b840f4b66d8e01078dca64) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.016996 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe30079045d437aa61bb5bfbfb96133 with delete_type TABLET_DATA_DELETED (replaced by tablet 18ea63263b9b466e922169eecafab747) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.018186 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 6926c49b787941d98c647543bd3b9ed5 (DEFAULT_TABLE table=default.Table_168a55a099724943a6e452febfeae7b3 [id=ecddff52c4604b1d99a6f1cd6fa37641]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.018620 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6926c49b787941d98c647543bd3b9ed5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:31.018819 32045 consensus_peers.cc:487] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 -> Peer 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313): Couldn't send request to peer 9752155c20ee42fbb7ff7b7d5b682444. Status: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 08:04:31.019059 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 87648b11b10b45fc9cc8c998bbceae92 (DEFAULT_TABLE table=default.Table_2fceb5254f914cd9a952ae0a805150c5 [id=ad88a05d1bbb448e94daa1471a373cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.019546 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87648b11b10b45fc9cc8c998bbceae92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.022845 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 36c8714cfbd84dd38cf5c0f305ef6843 (DEFAULT_TABLE table=default.Table_2092a828aea549a38d1bd42248c8804c [id=cce6d993b26f4f9ca197d807107e00cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.028851 952 ts_tablet_manager.cc:1939] T f3c850021e914530bb106ab7166be7c9 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:31.029387 32492 tablet_service.cc:1468] Processing CreateTablet for tablet a38eecf4a94d4536b98d4f6e037a0094 (DEFAULT_TABLE table=default.Table_e216bbb5ac2a4720b36183e30bd6ed91 [id=57e548bc818e4968b432cc3af298e675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.030056 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31a67b70cbc64bb3bfd5a34865109a01" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:31.030310 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7fcf583064f4538913d9157e93068dd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.030969 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c861e29e1e4e549021dae7255a856c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.031541 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32c82141edea4068b54755a85c6d7fb8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.031847 32515 raft_consensus.cc:2391] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:31.032167 32519 raft_consensus.cc:3058] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:31.033599 343 ts_tablet_manager.cc:1428] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.054s user 0.000s sys 0.002s
I20250623 08:04:31.035015 32517 raft_consensus.cc:2391] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:31.035058 952 tablet_replica.cc:331] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:31.035254 952 raft_consensus.cc:2241] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.035375 343 tablet_bootstrap.cc:492] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.035455 952 raft_consensus.cc:2270] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.036839 32519 raft_consensus.cc:2466] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:31.037575 952 ts_tablet_manager.cc:1905] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.039055 32349 tablet_service.cc:1468] Processing CreateTablet for tablet 7874a1d5cb9a464294ca32b2767efbf3 (DEFAULT_TABLE table=default.Table_8ec723069381477ca64f5aaad0df76fb [id=2a5a01b05a554ac1ad6abd21ad00d734]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.039515 32349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7874a1d5cb9a464294ca32b2767efbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.039775 343 tablet_bootstrap.cc:654] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.048821 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 36c8714cfbd84dd38cf5c0f305ef6843 (DEFAULT_TABLE table=default.Table_2092a828aea549a38d1bd42248c8804c [id=cce6d993b26f4f9ca197d807107e00cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.049407 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c8714cfbd84dd38cf5c0f305ef6843. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:31.050273 32054 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fd478c007914d2cb09d929b50322a64 consensus state has no leader
I20250623 08:04:31.053177 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_37287a2bae6a4d72ab9a957a92be554b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.053951 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37287a2bae6a4d72ab9a957a92be554b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.055431 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_303d632a02404fbe8b9baa353a8d3747"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:31.055707 32361 tablet_service.cc:1468] Processing CreateTablet for tablet a38eecf4a94d4536b98d4f6e037a0094 (DEFAULT_TABLE table=default.Table_e216bbb5ac2a4720b36183e30bd6ed91 [id=57e548bc818e4968b432cc3af298e675]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:31.056044 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_303d632a02404fbe8b9baa353a8d3747 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.056259 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38eecf4a94d4536b98d4f6e037a0094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.057839 32057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_463abc7f8321496da24ccb06cba92d25" } new_table_name: "default.Table_fe7f6ce28d01454f907e558368212098" modify_external_catalogs: true
I20250623 08:04:31.059383 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_ac62bae678de4b11a971f7bbccd50303"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.060231 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac62bae678de4b11a971f7bbccd50303 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.062831 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_2891f87970d64c6383f346e5b51e9ad3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.063503 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2891f87970d64c6383f346e5b51e9ad3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.066582 979 ts_tablet_manager.cc:1918] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:31.066733 979 log.cc:1199] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/965f1f0a107b4653b6aac01e0d8e259a
I20250623 08:04:31.071691 343 tablet_bootstrap.cc:492] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.071867 343 ts_tablet_manager.cc:1397] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.005s
I20250623 08:04:31.074646 343 raft_consensus.cc:357] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.076061 339 tablet_bootstrap.cc:492] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:31.076601 339 ts_tablet_manager.cc:1397] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.445s user 0.015s sys 0.019s
I20250623 08:04:31.076979 343 raft_consensus.cc:383] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.077517 343 raft_consensus.cc:738] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.072940 979 ts_tablet_manager.cc:1939] T 965f1f0a107b4653b6aac01e0d8e259a P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.082720 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0545daa6fba846adbc573a63d8dbcd66" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.083037 979 tablet_replica.cc:331] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.083030 32519 raft_consensus.cc:2466] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:31.083316 979 raft_consensus.cc:2241] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:31.083479 32043 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.60:39691, user_credentials={real_user=slave}} blocked reactor thread for 71494us
I20250623 08:04:31.083545 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7874a1d5cb9a464294ca32b2767efbf3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:31.084298 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0b2af698b6e4ed0ae66a9bb32be7b31 already in progress: creating tablet
I20250623 08:04:31.085775 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 965f1f0a107b4653b6aac01e0d8e259a (table default.Table_5f159a56c60144b99a6847ec6080623e [id=ec1f9e35e20c47fcbe2d4b80c7abeabc]) successfully deleted
I20250623 08:04:31.085922 981 log.cc:1199] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/4fe30079045d437aa61bb5bfbfb96133
I20250623 08:04:31.086987 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet c0f87fd62f1545089d00757f8214593a (table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]) successfully deleted
I20250623 08:04:31.087729 981 ts_tablet_manager.cc:1939] T 4fe30079045d437aa61bb5bfbfb96133 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:31.092300 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 2b405f87bb1e4d0686a1f627835d901f (DEFAULT_TABLE table=default.Table_f40b881c21294c10b35962b75a30524c [id=bdca6a821a6b48cbb3366d8912abeeda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.092806 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b405f87bb1e4d0686a1f627835d901f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.092852 979 raft_consensus.cc:2270] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.094570 343 consensus_queue.cc:260] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.095733 979 ts_tablet_manager.cc:1905] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.088812 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 4ac9f38877b24d619c280a1aaa0b7b08 (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]) successfully deleted
I20250623 08:04:31.099231 343 ts_tablet_manager.cc:1428] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250623 08:04:31.102821 343 tablet_bootstrap.cc:492] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.107970 32349 tablet_service.cc:1468] Processing CreateTablet for tablet 42c861e29e1e4e549021dae7255a856c (DEFAULT_TABLE table=default.Table_32db9f68dd974fb8aff372a8eeccce3d [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.109580 32349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c861e29e1e4e549021dae7255a856c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.119544 343 tablet_bootstrap.cc:654] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.120047 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36c8714cfbd84dd38cf5c0f305ef6843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.124208 979 ts_tablet_manager.cc:1918] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.124367 979 log.cc:1199] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/0311ce67989f49e8866234a88e5b4d09
I20250623 08:04:31.125732 979 ts_tablet_manager.cc:1939] T 0311ce67989f49e8866234a88e5b4d09 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.125882 32356 tablet_service.cc:1468] Processing CreateTablet for tablet f4a96ae9190640dea11bd11a9f0e4372 (DEFAULT_TABLE table=default.Table_0666030ad7ec4ef89335cdbaf0cca443 [id=bf6ca5aa8d5044868244a9b37cfa9519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.129159 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a38eecf4a94d4536b98d4f6e037a0094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.134178 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_35a9f061c58a4dd1b8ad6e194d32bbde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.134812 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35a9f061c58a4dd1b8ad6e194d32bbde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.134848 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417a050eada4403ebbddff255fb6fb5a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.134977 979 tablet_replica.cc:331] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.135205 979 raft_consensus.cc:2241] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.135728 979 raft_consensus.cc:2270] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.136227 32055 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ecfe5fe3793c43f6a76c51bad1437789
I20250623 08:04:31.136587 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_f66b3fa5edc14cffaa95a0a0a4f62f67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:31.137037 979 ts_tablet_manager.cc:1905] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:31.137147 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f66b3fa5edc14cffaa95a0a0a4f62f67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.138635 32053 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0242542a86ba40a28474c9832cd8ed3c" } modify_external_catalogs: true
I20250623 08:04:31.138842 32053 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0242542a86ba40a28474c9832cd8ed3c" } modify_external_catalogs: true
I20250623 08:04:31.142911 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adc3cf4355c24b7395fcb22af8520346" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:31.142932 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67c44697ecab4dd9b43f544a602897ca" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.143128 32650 raft_consensus.cc:3058] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:31.143317 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b81263a9ee9d4da1ac721c3b2bd21248" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:31.144277 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f3d32798168345edb9898751d083057c
I20250623 08:04:31.146111 32650 raft_consensus.cc:2466] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:31.146819 32589 leader_election.cc:304] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.147491 32589 leader_election.cc:304] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.147962 32589 leader_election.cc:304] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.148416 32589 leader_election.cc:304] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.148993 819 raft_consensus.cc:2747] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:31.149044 1011 raft_consensus.cc:2802] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:31.149200 819 raft_consensus.cc:2802] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:31.149296 945 raft_consensus.cc:2747] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:31.149319 819 raft_consensus.cc:491] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:31.149410 819 raft_consensus.cc:3058] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:31.149204 1011 raft_consensus.cc:695] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:31.150167 1011 consensus_queue.cc:237] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.150563 32059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_91b12dbe72c64d16a3334faf46445848" } modify_external_catalogs: true
I20250623 08:04:31.150736 32059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_91b12dbe72c64d16a3334faf46445848" } modify_external_catalogs: true
I20250623 08:04:31.152552 819 raft_consensus.cc:513] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.158720 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7fcf583064f4538913d9157e93068dd" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:31.160535 819 leader_election.cc:290] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:31.163489 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 0311ce67989f49e8866234a88e5b4d09 (table default.Table_e962b9945a5041039f7e33b98b653175 [id=9d9e232100264e378516c06c61310750]) successfully deleted
I20250623 08:04:31.164656 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet f3d32798168345edb9898751d083057c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.166877 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "358cb6ac24414bdfbc32155e732f9ac8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.167850 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44fd3c67185484e93450682364dc8a8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.168077 32515 raft_consensus.cc:2466] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
W20250623 08:04:31.168093 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 965f1f0a107b4653b6aac01e0d8e259a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 965f1f0a107b4653b6aac01e0d8e259a already in progress: creating tablet
W20250623 08:04:31.168385 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0b2af698b6e4ed0ae66a9bb32be7b31 already in progress: creating tablet
I20250623 08:04:31.168608 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.170471 32516 raft_consensus.cc:2391] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
W20250623 08:04:31.171329 1015 log.cc:927] Time spent T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250623 08:04:31.171872 32517 raft_consensus.cc:3058] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:31.169375 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7fcf583064f4538913d9157e93068dd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:31.172901 32513 raft_consensus.cc:2391] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:31.176311 657 raft_consensus.cc:491] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.176467 657 raft_consensus.cc:513] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.177093 657 leader_election.cc:290] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:31.177287 32518 raft_consensus.cc:2466] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:31.178488 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:31.184494 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet f3c850021e914530bb106ab7166be7c9 (table default.Table_fd49e04fcac14f0194cc9fed910601fc [id=17b055700da64cc1b19f93dd5edf40d5]) successfully deleted
I20250623 08:04:31.192101 979 ts_tablet_manager.cc:1918] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:31.192248 979 log.cc:1199] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/2060a643d0da47efa868f52bdb331b1f
I20250623 08:04:31.192680 979 ts_tablet_manager.cc:1939] T 2060a643d0da47efa868f52bdb331b1f P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.196978 979 tablet_replica.cc:331] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.197185 979 raft_consensus.cc:2241] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.197186 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 2060a643d0da47efa868f52bdb331b1f (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]) successfully deleted
I20250623 08:04:31.197376 979 raft_consensus.cc:2270] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.198385 979 ts_tablet_manager.cc:1905] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.199242 32517 raft_consensus.cc:2466] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:31.199529 32453 leader_election.cc:304] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:31.199776 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 43941.5us
I20250623 08:04:31.200207 657 raft_consensus.cc:2802] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:31.200407 657 raft_consensus.cc:695] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:31.200749 657 consensus_queue.cc:237] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.205009 952 ts_tablet_manager.cc:1918] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.212129 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a96ae9190640dea11bd11a9f0e4372. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.213769 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 6926c49b787941d98c647543bd3b9ed5 (DEFAULT_TABLE table=default.Table_168a55a099724943a6e452febfeae7b3 [id=ecddff52c4604b1d99a6f1cd6fa37641]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.213946 952 log.cc:1199] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/0311ce67989f49e8866234a88e5b4d09
I20250623 08:04:31.214462 952 ts_tablet_manager.cc:1939] T 0311ce67989f49e8866234a88e5b4d09 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:31.214778 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_b9220275c4064c11bb0618d6a459aab6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:31.215049 339 raft_consensus.cc:357] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.215245 952 tablet_replica.cc:331] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: stopping tablet replica
W20250623 08:04:31.215435 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9220275c4064c11bb0618d6a459aab6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.215550 952 raft_consensus.cc:2241] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:31.216710 32057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31a67b70cbc64bb3bfd5a34865109a01 consensus state has no leader
I20250623 08:04:31.216930 952 raft_consensus.cc:2270] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.217051 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_7202dd9a46994e949c2368ac8f51b53d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:31.217124 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0545daa6fba846adbc573a63d8dbcd66" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:31.217536 952 ts_tablet_manager.cc:1905] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:31.217567 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7202dd9a46994e949c2368ac8f51b53d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.217846 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6926c49b787941d98c647543bd3b9ed5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:31.220544 32589 consensus_peers.cc:487] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6a97b34e386443fa19a9601621b5017. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:31.220877 32589 leader_election.cc:343] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.221210 32054 master_service.cc:948] ReplaceTablet: received request to replace tablet 70f9d21b8c144f9cada9bdbe0fbb1538 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:31.222905 32050 master_service.cc:948] ReplaceTablet: received request to replace tablet 32c82141edea4068b54755a85c6d7fb8 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:31.224596 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_4c1c98c205e04f4e90e5a301451870da" } new_table_name: "default.Table_68acbd516e8f4990a2f197ac131bc6ba" modify_external_catalogs: true
I20250623 08:04:31.227483 874 leader_election.cc:290] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:31.227810 343 tablet_bootstrap.cc:492] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.228085 343 ts_tablet_manager.cc:1397] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.129s user 0.014s sys 0.000s
I20250623 08:04:31.229364 343 raft_consensus.cc:357] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.229663 343 raft_consensus.cc:383] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.229821 343 raft_consensus.cc:738] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.230226 343 consensus_queue.cc:260] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.231894 343 ts_tablet_manager.cc:1428] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:31.232950 343 tablet_bootstrap.cc:492] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.235152 343 tablet_bootstrap.cc:654] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.235860 981 tablet_replica.cc:331] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:31.243109 979 ts_tablet_manager.cc:1918] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:31.245349 979 log.cc:1199] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/4cb66c7a273a412db3ae77af9e30d036
I20250623 08:04:31.245895 979 ts_tablet_manager.cc:1939] T 4cb66c7a273a412db3ae77af9e30d036 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.246975 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 4cb66c7a273a412db3ae77af9e30d036 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]) successfully deleted
I20250623 08:04:31.247313 979 tablet_replica.cc:331] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.247781 979 raft_consensus.cc:2241] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.248291 979 raft_consensus.cc:2270] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.249141 979 ts_tablet_manager.cc:1905] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.254981 339 raft_consensus.cc:383] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.261705 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet f3d32798168345edb9898751d083057c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:31.262135 339 raft_consensus.cc:738] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.266271 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 5fdc9e53da264a5ca1f70e1846c906f0
I20250623 08:04:31.267066 32053 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_7b9c8fa18d93462685ad015a8cf62225" } new_table_name: "default.Table_6acb69304c424830a635f22f12343b31" modify_external_catalogs: true
I20250623 08:04:31.267458 343 tablet_bootstrap.cc:492] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.267616 343 ts_tablet_manager.cc:1397] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.003s
I20250623 08:04:31.268353 343 raft_consensus.cc:357] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.268545 343 raft_consensus.cc:383] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.268652 343 raft_consensus.cc:738] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.268788 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_6889774a49f94dddaef8079edda86978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:31.268903 343 consensus_queue.cc:260] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:31.269351 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6889774a49f94dddaef8079edda86978 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.269359 1001 consensus_queue.cc:1035] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20250623 08:04:31.269479 343 ts_tablet_manager.cc:1428] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:31.269835 343 tablet_bootstrap.cc:492] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.270093 339 consensus_queue.cc:260] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.271225 339 ts_tablet_manager.cc:1428] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.194s user 0.000s sys 0.003s
I20250623 08:04:31.273758 981 raft_consensus.cc:2241] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:31.275879 343 tablet_bootstrap.cc:654] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.277423 979 ts_tablet_manager.cc:1918] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:31.282887 979 log.cc:1199] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/279624b8bb834b95a4f08045968ebfb4
I20250623 08:04:31.284114 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]). Expected version 2 got 0
I20250623 08:04:31.284587 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:31.284957 32055 catalog_manager.cc:5582] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:31.283938 979 ts_tablet_manager.cc:1939] T 279624b8bb834b95a4f08045968ebfb4 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.285632 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c7fcf583064f4538913d9157e93068dd (table default.Table_a65e08da5dbe4b408014e6c3b662450f [id=abda89a30971466ea9a1f410d424ef58]). Expected version 1 got 0
I20250623 08:04:31.286008 32055 catalog_manager.cc:5582] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:31.286370 979 tablet_replica.cc:331] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.286470 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 31a67b70cbc64bb3bfd5a34865109a01 (table default.Table_fe7f6ce28d01454f907e558368212098 [id=7a569a9fe3bd4c608ddca48a734df1da]). Expected version 1 got 0
I20250623 08:04:31.286630 979 raft_consensus.cc:2241] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.286865 979 raft_consensus.cc:2270] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.286895 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 0861107ab4624a05bd7a4d24b420c502 (table default.Table_977fe8fc665f4267b394065e22f67eea [id=be9754ecb40a413d9940224adbd7e2eb]). Expected version 2 got 0
I20250623 08:04:31.287194 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet ba3d2fa01d574dd992dc904ac024ba3a (table default.Table_d496fd69b6f140b88b5abc86c8839ee2 [id=6c57db68f5ed45a3b64d1e3a02ce8a2a]). Expected version 1 got 0
I20250623 08:04:31.287523 979 ts_tablet_manager.cc:1905] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.287654 32055 catalog_manager.cc:5582] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:31.292575 343 tablet_bootstrap.cc:492] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.292788 343 ts_tablet_manager.cc:1397] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20250623 08:04:31.293589 343 raft_consensus.cc:357] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.293823 343 raft_consensus.cc:383] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.293980 343 raft_consensus.cc:738] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.294301 343 consensus_queue.cc:260] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.295023 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9e53da264a5ca1f70e1846c906f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.295848 343 ts_tablet_manager.cc:1428] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:31.296190 343 tablet_bootstrap.cc:492] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.296130 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 771ecfc41e2b4007a4d213cec5b128c6 (DEFAULT_TABLE table=default.Table_bf77bbe8745846d3b962bd5b7fed73ce [id=2158e340258845d19f99a6300ec4c86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.296609 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 771ecfc41e2b4007a4d213cec5b128c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.297140 32623 tablet_service.cc:1468] Processing CreateTablet for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (DEFAULT_TABLE table=default.Table_f5479d2204fd477ba893d16a0f77c4fa [id=bee83212a54d4711ad16ad829e99470b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.297382 896 consensus_queue.cc:1035] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250623 08:04:31.297505 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73453eb1a0394cfe8d49adcc71d4f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:31.302183 746 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.007s sys 0.005s
W20250623 08:04:31.302387 746 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s user 0.007s sys 0.005s
I20250623 08:04:31.302522 343 tablet_bootstrap.cc:654] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.303303 339 tablet_bootstrap.cc:492] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:31.311041 979 ts_tablet_manager.cc:1918] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250623 08:04:31.311779 32314 consensus_peers.cc:487] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae7c2ac546114a819a543f1667fb7844. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:31.312637 979 log.cc:1199] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/19536c16ff32460e9db571e4ce411bab
I20250623 08:04:31.313282 979 ts_tablet_manager.cc:1939] T 19536c16ff32460e9db571e4ce411bab P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.315369 32623 tablet_service.cc:1468] Processing CreateTablet for tablet b84b8b8713774ab4896842feb9fb075e (DEFAULT_TABLE table=default.Table_fcc07b378bf64122ae65bc17f130b2f0 [id=cb527c3a47d74b459b377b64b23f7675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.315835 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84b8b8713774ab4896842feb9fb075e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.316825 979 tablet_replica.cc:331] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.324076 981 pending_rounds.cc:70] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:31.327731 979 raft_consensus.cc:2241] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.328025 979 raft_consensus.cc:2270] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.328796 979 ts_tablet_manager.cc:1905] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.327425 343 tablet_bootstrap.cc:492] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.329092 343 ts_tablet_manager.cc:1397] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20250623 08:04:31.314868 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 279624b8bb834b95a4f08045968ebfb4 (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]) successfully deleted
I20250623 08:04:31.329960 343 raft_consensus.cc:357] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.330245 343 raft_consensus.cc:383] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.330384 343 raft_consensus.cc:738] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.330715 343 consensus_queue.cc:260] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.330502 981 pending_rounds.cc:77] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7170727407478296576 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "4ac9f38877b24d619c280a1aaa0b7b08" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 5 new_table_name: "default.Table_0ac52a7e208745e980f3a49c46b3cdb8" dest_uuid: "7351aa31800245659edc9837e27daffd" new_extra_config { } }
I20250623 08:04:31.332020 343 ts_tablet_manager.cc:1428] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:31.332692 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:31.333022 343 tablet_bootstrap.cc:492] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.333217 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 358cb6ac24414bdfbc32155e732f9ac8 (table default.Table_06b64498b6e148d5b86187f2ce7e4c31 [id=b827ea08f0b14c9aa2b2c3d02d15f150]). Expected version 1 got 0
I20250623 08:04:31.324852 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 (DEFAULT_TABLE table=default.Table_04e85a0dbf974a05a94b9a9713fcb3f2 [id=7febcca3a06346738e869f73028a972e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:31.334882 981 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35542: Aborted: Op aborted
I20250623 08:04:31.335531 981 raft_consensus.cc:2270] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:31.337709 1052 log.cc:927] Time spent T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250623 08:04:31.337958 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c9d1c8a67e149478a0d69842a4eb5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.341216 343 tablet_bootstrap.cc:654] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.346484 979 ts_tablet_manager.cc:1918] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.347087 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 19536c16ff32460e9db571e4ce411bab (table default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030 [id=f8961c82f9384b0db128a791e00351f6]) successfully deleted
I20250623 08:04:31.349936 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:31.347268 1012 raft_consensus.cc:491] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.353863 1012 raft_consensus.cc:513] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:31.354282 32050 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ecb422550efd40cf82bc92fca5c25eda consensus state has no leader
I20250623 08:04:31.354548 1012 leader_election.cc:290] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:31.347951 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 789d1c20aa72480ab71123c799bbde4c (DEFAULT_TABLE table=default.Table_da00c692e65345c496f4e52c106aa2e6 [id=5ea293756d514f06b8d749c615417969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.354673 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_da3ed120aff049bc85b20e2200850ae3" } new_table_name: "default.Table_e0696987af214f9fb46f9c76b529fca5" modify_external_catalogs: true
I20250623 08:04:31.355047 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 789d1c20aa72480ab71123c799bbde4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.356262 32057 master_service.cc:948] ReplaceTablet: received request to replace tablet 52d0a356c74f4daea9093c998b4bf73e from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:31.358326 32056 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_bc69503004f74bf59fd961b4f209baa6" } new_table_name: "default.Table_e5d470e2745643ee95ca5c01126dee26" modify_external_catalogs: true
I20250623 08:04:31.358807 981 ts_tablet_manager.cc:1905] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.349030 32618 tablet_service.cc:1468] Processing CreateTablet for tablet f8a5e35c1974456b8079ab8907c2652e (DEFAULT_TABLE table=default.Table_ee8a2c2391624131a4aac14a90311aa0 [id=c63ada5e02d74e049460990450ab4962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.349462 32624 tablet_service.cc:1468] Processing CreateTablet for tablet b0d1ec9d56ba40128325a50f204fe920 (DEFAULT_TABLE table=default.Table_3188d5c072e04adea5dae4c5ec7a71c8 [id=3e41f8259c5648c59b388ea9f4de99d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.349849 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (DEFAULT_TABLE table=default.Table_c76a280c52214058b9b4b2691090cc1f [id=ad7210d2665f463c8591e79cf62d15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.350185 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 45293db3c40b4e4194445f12e7475e66 (DEFAULT_TABLE table=default.Table_e8864c4ee64b4f4dbf04caa321024840 [id=537cece72a774ac99261a4c2a2e52c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.360589 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a5e35c1974456b8079ab8907c2652e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.360814 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af1dea3c6ca4f498ce6c039d72a14b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.363794 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 4fe30079045d437aa61bb5bfbfb96133 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]) successfully deleted
I20250623 08:04:31.364230 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45293db3c40b4e4194445f12e7475e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.367230 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d1ec9d56ba40128325a50f204fe920. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:31.369694 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet c0f87fd62f1545089d00757f8214593a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0f87fd62f1545089d00757f8214593a already in progress: creating tablet
I20250623 08:04:31.372339 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 37b38cb54d6c45308358f556a692eabe (DEFAULT_TABLE table=default.Table_c4f705a7cbd940da941c460b61c5dece [id=5d3fcb492ac246419c43b0618b4d797d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.372781 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b38cb54d6c45308358f556a692eabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.374564 979 log.cc:1199] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/1e47b10f46b8402e94354ebca573c194
I20250623 08:04:31.375370 343 tablet_bootstrap.cc:492] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.375527 343 ts_tablet_manager.cc:1397] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.001s
I20250623 08:04:31.377135 32634 tablet_service.cc:1468] Processing CreateTablet for tablet f9f4829e103b460eb582beff3b966bb4 (DEFAULT_TABLE table=default.Table_c300ef24f71d4bebaa367db14adb1b8a [id=5242fe018e4a4a13a365961197786944]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.377537 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9f4829e103b460eb582beff3b966bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.378021 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 306c4021b5d94a4996de07c222a8bfd8 (DEFAULT_TABLE table=default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.378355 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306c4021b5d94a4996de07c222a8bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.376448 343 raft_consensus.cc:357] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.385771 343 raft_consensus.cc:383] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.385883 343 raft_consensus.cc:738] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.374519 32618 tablet_service.cc:1468] Processing CreateTablet for tablet a8de2b1105564fdab386442b1de4cedb (DEFAULT_TABLE table=default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.387552 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8de2b1105564fdab386442b1de4cedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.389071 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 501b5233522f43c7aed3c9770adb5c03 (DEFAULT_TABLE table=default.Table_fbb1fc5910454112b795f72c26f1af5b [id=6d6ee37c928745ef877266573fe021f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.389528 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501b5233522f43c7aed3c9770adb5c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.360656 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f9e0d1b85a94ae9ae86e53688a55db5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:31.390821 32589 leader_election.cc:343] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.386075 343 consensus_queue.cc:260] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.394289 343 ts_tablet_manager.cc:1428] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20250623 08:04:31.394665 343 tablet_bootstrap.cc:492] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:31.397039 32044 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 31a67b70cbc64bb3bfd5a34865109a01 (table default.Table_fe7f6ce28d01454f907e558368212098 [id=7a569a9fe3bd4c608ddca48a734df1da], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 31a67b70cbc64bb3bfd5a34865109a01 consensus state has no leader
I20250623 08:04:31.397704 32623 tablet_service.cc:1468] Processing CreateTablet for tablet 9049e3d9b39f420bb51221808409fc54 (DEFAULT_TABLE table=default.Table_b7a92f8a00ba4931827eaee759bc8f91 [id=6fd0a78c429c40f8aa3b86e61e83a142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.400504 343 tablet_bootstrap.cc:654] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.401010 979 ts_tablet_manager.cc:1939] T 1e47b10f46b8402e94354ebca573c194 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:31.404911 32053 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f83f68ebda7e4826bbb17278cd4ac5a8 consensus state has no leader
I20250623 08:04:31.405364 32053 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_168a55a099724943a6e452febfeae7b3" } modify_external_catalogs: true
I20250623 08:04:31.405567 32053 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_168a55a099724943a6e452febfeae7b3" } modify_external_catalogs: true
I20250623 08:04:31.416245 342 tablet_bootstrap.cc:492] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:31.416744 342 ts_tablet_manager.cc:1397] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.490s user 0.026s sys 0.047s
I20250623 08:04:31.419518 1058 tablet.cc:1722] T 0861107ab4624a05bd7a4d24b420c502 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:31.420826 342 raft_consensus.cc:357] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.421046 342 raft_consensus.cc:383] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.423048 342 raft_consensus.cc:738] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.423332 342 consensus_queue.cc:260] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.423942 342 ts_tablet_manager.cc:1428] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250623 08:04:31.424263 342 tablet_bootstrap.cc:492] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:31.424803 32632 tablet_service.cc:1468] Processing CreateTablet for tablet f83f68ebda7e4826bbb17278cd4ac5a8 (DEFAULT_TABLE table=default.Table_6acb69304c424830a635f22f12343b31 [id=c1782341d0d34662889cff00d557ff81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.426126 342 tablet_bootstrap.cc:654] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.431377 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 38ecd4c243b840f4b66d8e01078dca64
I20250623 08:04:31.431977 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 70f9d21b8c144f9cada9bdbe0fbb1538
I20250623 08:04:31.432379 32054 catalog_manager.cc:6652] ReplaceTablet: tablet 70f9d21b8c144f9cada9bdbe0fbb1538 deleted and replaced by tablet db81250d1e164a7d94affd764fc0ff0c
W20250623 08:04:31.437476 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 225941us
I20250623 08:04:31.440213 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f
I20250623 08:04:31.442852 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 6926c49b787941d98c647543bd3b9ed5 (DEFAULT_TABLE table=default.Table_168a55a099724943a6e452febfeae7b3 [id=ecddff52c4604b1d99a6f1cd6fa37641]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.446861 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9e53da264a5ca1f70e1846c906f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:31.457240 1058 tablet.cc:1722] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:31.457677 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet f3d32798168345edb9898751d083057c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:31.463277 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6926c49b787941d98c647543bd3b9ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.463310 819 raft_consensus.cc:491] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.463476 819 raft_consensus.cc:513] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:31.464861 819 leader_election.cc:290] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:31.473606 32488 tablet_service.cc:1468] Processing CreateTablet for tablet 771ecfc41e2b4007a4d213cec5b128c6 (DEFAULT_TABLE table=default.Table_bf77bbe8745846d3b962bd5b7fed73ce [id=2158e340258845d19f99a6300ec4c86d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:31.475136 32589 leader_election.cc:343] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 16d6b1e81e4d438f85d8fee4dd7d8b7d
I20250623 08:04:31.475521 342 tablet_bootstrap.cc:492] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:31.389155 339 tablet_bootstrap.cc:654] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.478984 610 raft_consensus.cc:491] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.479190 610 raft_consensus.cc:513] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.479861 343 tablet_bootstrap.cc:492] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.479918 610 leader_election.cc:290] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:31.480029 343 ts_tablet_manager.cc:1397] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.000s
I20250623 08:04:31.480792 343 raft_consensus.cc:357] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.480969 343 raft_consensus.cc:383] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.481034 343 raft_consensus.cc:738] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.481243 343 consensus_queue.cc:260] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.483249 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9049e3d9b39f420bb51221808409fc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.485929 343 ts_tablet_manager.cc:1428] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250623 08:04:31.486233 343 tablet_bootstrap.cc:492] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.488010 343 tablet_bootstrap.cc:654] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:31.490221 32043 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.60:39691, user_credentials={real_user=slave}} blocked reactor thread for 69772us
I20250623 08:04:31.491403 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 with delete_type TABLET_DATA_DELETED (replaced by tablet db81250d1e164a7d94affd764fc0ff0c) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.491693 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 38ecd4c243b840f4b66d8e01078dca64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.491930 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.493157 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_e026a79cbd6c44efae4e91ab3db8e5fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.493892 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e026a79cbd6c44efae4e91ab3db8e5fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.495697 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_9c6bc2fbfc274d2e8414205f629cb439"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.496289 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c6bc2fbfc274d2e8414205f629cb439 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.500257 32059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fe7f6ce28d01454f907e558368212098" } new_table_name: "default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e" modify_external_catalogs: true
I20250623 08:04:31.501051 819 raft_consensus.cc:491] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.501219 819 raft_consensus.cc:513] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.501742 32058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_4550e74cc6e845719c1466d3acd1b723" } modify_external_catalogs: true
I20250623 08:04:31.501827 819 leader_election.cc:290] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:31.501924 32058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_4550e74cc6e845719c1466d3acd1b723" } modify_external_catalogs: true
I20250623 08:04:31.506407 1012 raft_consensus.cc:491] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.506579 1012 raft_consensus.cc:513] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.507254 1012 leader_election.cc:290] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:31.510133 979 tablet_replica.cc:331] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.510334 979 raft_consensus.cc:2241] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.510511 979 pending_rounds.cc:70] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:31.510567 979 pending_rounds.cc:77] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727401814892544 op_type: NO_OP noop_request { }
I20250623 08:04:31.510680 979 raft_consensus.cc:2887] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:31.510795 979 raft_consensus.cc:2270] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.511162 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83f68ebda7e4826bbb17278cd4ac5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.512293 342 ts_tablet_manager.cc:1397] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.010s
I20250623 08:04:31.512787 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 771ecfc41e2b4007a4d213cec5b128c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.514366 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (DEFAULT_TABLE table=default.Table_f5479d2204fd477ba893d16a0f77c4fa [id=bee83212a54d4711ad16ad829e99470b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.514876 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73453eb1a0394cfe8d49adcc71d4f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.516476 979 ts_tablet_manager.cc:1905] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.523954 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38ecd4c243b840f4b66d8e01078dca64" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:31.524636 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20e7a78cfc384033810603316b9e66bf" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:31.526445 979 ts_tablet_manager.cc:1918] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.527736 979 log.cc:1199] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/417a050eada4403ebbddff255fb6fb5a
I20250623 08:04:31.528224 979 ts_tablet_manager.cc:1939] T 417a050eada4403ebbddff255fb6fb5a P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.527573 343 tablet_bootstrap.cc:492] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.528496 343 ts_tablet_manager.cc:1397] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.001s
I20250623 08:04:31.529278 343 raft_consensus.cc:357] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.529536 343 raft_consensus.cc:383] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.529673 343 raft_consensus.cc:738] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.529963 343 consensus_queue.cc:260] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.531368 343 ts_tablet_manager.cc:1428] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:31.532325 343 tablet_bootstrap.cc:492] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.532321 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 (DEFAULT_TABLE table=default.Table_04e85a0dbf974a05a94b9a9713fcb3f2 [id=7febcca3a06346738e869f73028a972e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.532847 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c9d1c8a67e149478a0d69842a4eb5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.532339 342 raft_consensus.cc:357] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.533993 343 tablet_bootstrap.cc:654] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.513581 32498 tablet_service.cc:1468] Processing CreateTablet for tablet f4a96ae9190640dea11bd11a9f0e4372 (DEFAULT_TABLE table=default.Table_0666030ad7ec4ef89335cdbaf0cca443 [id=bf6ca5aa8d5044868244a9b37cfa9519]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:31.535673 32589 leader_election.cc:343] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:31.536907 32589 leader_election.cc:343] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.536630 979 tablet_replica.cc:331] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.539052 979 raft_consensus.cc:2241] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.540289 979 pending_rounds.cc:70] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:31.540575 979 pending_rounds.cc:77] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727401051365376 op_type: NO_OP noop_request { }
I20250623 08:04:31.540831 979 raft_consensus.cc:2887] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:31.541066 979 raft_consensus.cc:2270] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.541204 342 raft_consensus.cc:383] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.541324 342 raft_consensus.cc:738] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.541512 32492 tablet_service.cc:1468] Processing CreateTablet for tablet b84b8b8713774ab4896842feb9fb075e (DEFAULT_TABLE table=default.Table_fcc07b378bf64122ae65bc17f130b2f0 [id=cb527c3a47d74b459b377b64b23f7675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.542181 979 ts_tablet_manager.cc:1905] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:31.549851 32455 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250623 08:04:31.543835 343 tablet_bootstrap.cc:492] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.551311 343 ts_tablet_manager.cc:1397] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
I20250623 08:04:31.550282 342 consensus_queue.cc:260] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.552919 343 raft_consensus.cc:357] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.553789 343 raft_consensus.cc:383] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.553974 343 raft_consensus.cc:738] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.554284 343 consensus_queue.cc:260] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.554407 32495 tablet_service.cc:1468] Processing CreateTablet for tablet f8a5e35c1974456b8079ab8907c2652e (DEFAULT_TABLE table=default.Table_ee8a2c2391624131a4aac14a90311aa0 [id=c63ada5e02d74e049460990450ab4962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.554950 342 ts_tablet_manager.cc:1428] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.041s user 0.003s sys 0.000s
I20250623 08:04:31.555660 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (DEFAULT_TABLE table=default.Table_c76a280c52214058b9b4b2691090cc1f [id=ad7210d2665f463c8591e79cf62d15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.555779 343 ts_tablet_manager.cc:1428] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250623 08:04:31.555845 32493 tablet_service.cc:1468] Processing CreateTablet for tablet b0d1ec9d56ba40128325a50f204fe920 (DEFAULT_TABLE table=default.Table_3188d5c072e04adea5dae4c5ec7a71c8 [id=3e41f8259c5648c59b388ea9f4de99d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.555153 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 789d1c20aa72480ab71123c799bbde4c (DEFAULT_TABLE table=default.Table_da00c692e65345c496f4e52c106aa2e6 [id=5ea293756d514f06b8d749c615417969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.556100 343 tablet_bootstrap.cc:492] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.558866 343 tablet_bootstrap.cc:654] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.572446 979 ts_tablet_manager.cc:1918] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.572690 979 log.cc:1199] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/175ee90133994154b7e4b58b0770db3b
I20250623 08:04:31.573141 979 ts_tablet_manager.cc:1939] T 175ee90133994154b7e4b58b0770db3b P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.574532 979 tablet_replica.cc:331] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.574810 979 raft_consensus.cc:2241] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.575034 979 pending_rounds.cc:70] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:31.576109 979 pending_rounds.cc:77] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727402781786112 op_type: NO_OP noop_request { }
I20250623 08:04:31.576323 979 raft_consensus.cc:2887] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:31.576510 979 raft_consensus.cc:2270] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.576020 343 tablet_bootstrap.cc:492] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.576900 343 ts_tablet_manager.cc:1397] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250623 08:04:31.577783 343 raft_consensus.cc:357] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.578022 343 raft_consensus.cc:383] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.578151 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 6926c49b787941d98c647543bd3b9ed5
I20250623 08:04:31.578157 343 raft_consensus.cc:738] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.578469 343 consensus_queue.cc:260] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:31.578647 32050 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 52d0a356c74f4daea9093c998b4bf73e consensus state has no leader
I20250623 08:04:31.579212 343 ts_tablet_manager.cc:1428] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:31.579299 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 52d0a356c74f4daea9093c998b4bf73e
I20250623 08:04:31.579810 32057 catalog_manager.cc:6652] ReplaceTablet: tablet 52d0a356c74f4daea9093c998b4bf73e deleted and replaced by tablet bb06a340f82047ac8ec3516edb08f4b6
I20250623 08:04:31.580235 343 tablet_bootstrap.cc:492] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:31.580471 32056 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 712abfca1fa4489f882534c33b9db1be consensus state has no leader
I20250623 08:04:31.582580 343 tablet_bootstrap.cc:654] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.583843 979 ts_tablet_manager.cc:1905] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.593237 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 1e47b10f46b8402e94354ebca573c194 (table default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]) successfully deleted
I20250623 08:04:31.594687 32487 tablet_service.cc:1468] Processing CreateTablet for tablet 45293db3c40b4e4194445f12e7475e66 (DEFAULT_TABLE table=default.Table_e8864c4ee64b4f4dbf04caa321024840 [id=537cece72a774ac99261a4c2a2e52c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.595963 343 tablet_bootstrap.cc:492] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.596153 343 ts_tablet_manager.cc:1397] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250623 08:04:31.597065 343 raft_consensus.cc:357] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.597277 343 raft_consensus.cc:383] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.597371 343 raft_consensus.cc:738] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.597653 343 consensus_queue.cc:260] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.599074 343 ts_tablet_manager.cc:1428] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:31.599948 343 tablet_bootstrap.cc:492] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.600404 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a96ae9190640dea11bd11a9f0e4372. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.601745 343 tablet_bootstrap.cc:654] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.603004 1001 raft_consensus.cc:491] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.603564 979 ts_tablet_manager.cc:1918] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.603713 979 log.cc:1199] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/67c44697ecab4dd9b43f544a602897ca
I20250623 08:04:31.603777 342 tablet_bootstrap.cc:492] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:31.604131 979 ts_tablet_manager.cc:1939] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.606498 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adc3cf4355c24b7395fcb22af8520346" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:31.608989 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:31.609170 32589 leader_election.cc:343] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.609205 32653 raft_consensus.cc:2466] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:31.609326 32589 leader_election.cc:304] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.609748 32589 leader_election.cc:304] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.610144 32589 leader_election.cc:304] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.610204 819 raft_consensus.cc:2747] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:31.610241 944 raft_consensus.cc:2802] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:31.610357 944 raft_consensus.cc:491] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:31.610460 944 raft_consensus.cc:3058] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:31.614758 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20e7a78cfc384033810603316b9e66bf" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.614957 944 raft_consensus.cc:513] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:31.615201 32589 raft_consensus.cc:2684] T 67c44697ecab4dd9b43f544a602897ca P 125745849172449ba4be8244d61fdcb3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:31.616030 32589 leader_election.cc:304] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:31.616269 32453 consensus_peers.cc:487] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 175ee90133994154b7e4b58b0770db3b. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:31.616911 32589 consensus_peers.cc:487] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:31.617673 710 raft_consensus.cc:2802] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:31.617887 710 raft_consensus.cc:491] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:31.618014 710 raft_consensus.cc:3058] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:31.618196 32589 leader_election.cc:343] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 16d6b1e81e4d438f85d8fee4dd7d8b7d
I20250623 08:04:31.618340 32589 leader_election.cc:304] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.601178 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_a7cbd31e5c664fba885a13dcbd9105ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:31.619148 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a5e35c1974456b8079ab8907c2652e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:31.619547 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7cbd31e5c664fba885a13dcbd9105ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.619657 1011 raft_consensus.cc:2747] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:31.601230 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_eda413af2cf84c009c8fc4b375b57e2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.621320 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eda413af2cf84c009c8fc4b375b57e2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.621536 710 raft_consensus.cc:513] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.622129 710 leader_election.cc:290] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:31.601708 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_38be2bdc4e8f4cd7be412f6d1b883585"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.622952 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38be2bdc4e8f4cd7be412f6d1b883585 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.617632 944 leader_election.cc:290] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:31.601792 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_4533668b8daa4ff3ad00ccc95e97e032"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.625348 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4533668b8daa4ff3ad00ccc95e97e032 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.630501 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 306c4021b5d94a4996de07c222a8bfd8 (DEFAULT_TABLE table=default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.630856 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 37b38cb54d6c45308358f556a692eabe (DEFAULT_TABLE table=default.Table_c4f705a7cbd940da941c460b61c5dece [id=5d3fcb492ac246419c43b0618b4d797d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.633176 32481 tablet_service.cc:1515] Processing DeleteTablet for tablet 38ecd4c243b840f4b66d8e01078dca64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:31.633821 1021 raft_consensus.cc:491] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.633865 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0545daa6fba846adbc573a63d8dbcd66" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:31.633997 1021 raft_consensus.cc:513] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.634126 32514 raft_consensus.cc:3058] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:31.634881 1021 leader_election.cc:290] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:31.635315 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f9e0d1b85a94ae9ae86e53688a55db5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.635499 32513 raft_consensus.cc:2466] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:31.635753 657 raft_consensus.cc:491] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.635731 1062 raft_consensus.cc:491] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:31.635872 657 raft_consensus.cc:513] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.636489 657 leader_election.cc:290] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:31.636932 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38ecd4c243b840f4b66d8e01078dca64" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.637136 32518 raft_consensus.cc:2466] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:31.640818 952 ts_tablet_manager.cc:1918] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.641698 32490 tablet_service.cc:1468] Processing CreateTablet for tablet f9f4829e103b460eb582beff3b966bb4 (DEFAULT_TABLE table=default.Table_c300ef24f71d4bebaa367db14adb1b8a [id=5242fe018e4a4a13a365961197786944]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.642154 32453 leader_election.cc:304] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:31.644631 32494 tablet_service.cc:1468] Processing CreateTablet for tablet a8de2b1105564fdab386442b1de4cedb (DEFAULT_TABLE table=default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.643219 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 with delete_type TABLET_DATA_DELETED (replaced by tablet db81250d1e164a7d94affd764fc0ff0c) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:31.646202 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 501b5233522f43c7aed3c9770adb5c03 (DEFAULT_TABLE table=default.Table_fbb1fc5910454112b795f72c26f1af5b [id=6d6ee37c928745ef877266573fe021f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.646930 32483 tablet_service.cc:1468] Processing CreateTablet for tablet f83f68ebda7e4826bbb17278cd4ac5a8 (DEFAULT_TABLE table=default.Table_6acb69304c424830a635f22f12343b31 [id=c1782341d0d34662889cff00d557ff81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.647854 32484 tablet_service.cc:1468] Processing CreateTablet for tablet 9049e3d9b39f420bb51221808409fc54 (DEFAULT_TABLE table=default.Table_b7a92f8a00ba4931827eaee759bc8f91 [id=6fd0a78c429c40f8aa3b86e61e83a142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.602167 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a25bdc9b7e48467fba3771365e58bcbc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:31.655157 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44fd3c67185484e93450682364dc8a8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:31.655114 1085 tablet.cc:1722] T 0861107ab4624a05bd7a4d24b420c502 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:31.656661 1062 raft_consensus.cc:513] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:31.657527 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:31.659650 1087 tablet.cc:1722] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:31.661301 343 tablet_bootstrap.cc:492] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:31.661523 343 ts_tablet_manager.cc:1397] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.001s
I20250623 08:04:31.662406 343 raft_consensus.cc:357] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.666448 343 raft_consensus.cc:383] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:31.666646 343 raft_consensus.cc:738] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:31.666978 343 consensus_queue.cc:260] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.667662 343 ts_tablet_manager.cc:1428] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:31.682566 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:31.682099 952 log.cc:1199] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/c6a97b34e386443fa19a9601621b5017
I20250623 08:04:31.673830 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 417a050eada4403ebbddff255fb6fb5a (table default.Table_24b73ca48b554c6698827e520b13b121 [id=2857dcb54e8b4109996cea363085a395]) successfully deleted
I20250623 08:04:31.686311 32487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45293db3c40b4e4194445f12e7475e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.686654 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af1dea3c6ca4f498ce6c039d72a14b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.687631 32514 raft_consensus.cc:2466] T 0545daa6fba846adbc573a63d8dbcd66 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:31.687790 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 789d1c20aa72480ab71123c799bbde4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.688081 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8de2b1105564fdab386442b1de4cedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.688319 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84b8b8713774ab4896842feb9fb075e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.688504 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9f4829e103b460eb582beff3b966bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.689229 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d1ec9d56ba40128325a50f204fe920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.689529 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83f68ebda7e4826bbb17278cd4ac5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.692852 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 175ee90133994154b7e4b58b0770db3b (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]) successfully deleted
I20250623 08:04:31.694257 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9049e3d9b39f420bb51221808409fc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.695631 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501b5233522f43c7aed3c9770adb5c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.697677 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306c4021b5d94a4996de07c222a8bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.698135 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b38cb54d6c45308358f556a692eabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.697242 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 67c44697ecab4dd9b43f544a602897ca (table default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]) successfully deleted
I20250623 08:04:31.700142 1001 raft_consensus.cc:513] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:31.700717 32589 leader_election.cc:343] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:31.701082 32589 leader_election.cc:343] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:31.702437 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet a414144276f6484ea4690204752e2142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a414144276f6484ea4690204752e2142 already in progress: creating tablet
I20250623 08:04:31.703490 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7fcf583064f4538913d9157e93068dd" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:31.703739 32381 raft_consensus.cc:2391] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:31.704732 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 6926c49b787941d98c647543bd3b9ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.705243 339 tablet_bootstrap.cc:492] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:31.705518 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 52d0a356c74f4daea9093c998b4bf73e with delete_type TABLET_DATA_DELETED (replaced by tablet bb06a340f82047ac8ec3516edb08f4b6) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.706131 339 ts_tablet_manager.cc:1397] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.435s user 0.012s sys 0.003s
I20250623 08:04:31.708545 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 0311ce67989f49e8866234a88e5b4d09 (table default.Table_e962b9945a5041039f7e33b98b653175 [id=9d9e232100264e378516c06c61310750]) successfully deleted
W20250623 08:04:31.708812 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet f4a0b9402a264800a1ea39707c10da0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4a0b9402a264800a1ea39707c10da0c already in progress: creating tablet
I20250623 08:04:31.709004 1062 leader_election.cc:290] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:31.709611 1069 log.cc:927] Time spent T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250623 08:04:31.709915 979 tablet_replica.cc:331] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.710152 979 raft_consensus.cc:2241] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.710350 979 raft_consensus.cc:2270] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.710937 979 ts_tablet_manager.cc:1905] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.709025 1001 leader_election.cc:290] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:31.715070 343 tablet_bootstrap.cc:492] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:31.715725 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 501b5233522f43c7aed3c9770adb5c03 (DEFAULT_TABLE table=default.Table_fbb1fc5910454112b795f72c26f1af5b [id=6d6ee37c928745ef877266573fe021f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.716140 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501b5233522f43c7aed3c9770adb5c03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.716316 32344 tablet_service.cc:1515] Processing DeleteTablet for tablet 38ecd4c243b840f4b66d8e01078dca64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:31.716679 32344 tablet_service.cc:1515] Processing DeleteTablet for tablet 6926c49b787941d98c647543bd3b9ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:31.716785 32356 tablet_service.cc:1468] Processing CreateTablet for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (DEFAULT_TABLE table=default.Table_f5479d2204fd477ba893d16a0f77c4fa [id=bee83212a54d4711ad16ad829e99470b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.716892 32344 tablet_service.cc:1515] Processing DeleteTablet for tablet 52d0a356c74f4daea9093c998b4bf73e with delete_type TABLET_DATA_DELETED (replaced by tablet bb06a340f82047ac8ec3516edb08f4b6) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:31.717207 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73453eb1a0394cfe8d49adcc71d4f3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.718225 32343 tablet_service.cc:1515] Processing DeleteTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 with delete_type TABLET_DATA_DELETED (replaced by tablet db81250d1e164a7d94affd764fc0ff0c) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:31.718942 343 tablet_bootstrap.cc:654] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:31.721017 979 ts_tablet_manager.cc:1918] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.721171 979 log.cc:1199] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/7da84969a6464b5caeaf47af0cbaaa4d
I20250623 08:04:31.721689 979 ts_tablet_manager.cc:1939] T 7da84969a6464b5caeaf47af0cbaaa4d P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.721674 32348 tablet_service.cc:1468] Processing CreateTablet for tablet b84b8b8713774ab4896842feb9fb075e (DEFAULT_TABLE table=default.Table_fcc07b378bf64122ae65bc17f130b2f0 [id=cb527c3a47d74b459b377b64b23f7675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.722036 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b84b8b8713774ab4896842feb9fb075e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.726404 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 789d1c20aa72480ab71123c799bbde4c (DEFAULT_TABLE table=default.Table_da00c692e65345c496f4e52c106aa2e6 [id=5ea293756d514f06b8d749c615417969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.727058 32351 tablet_service.cc:1468] Processing CreateTablet for tablet f8a5e35c1974456b8079ab8907c2652e (DEFAULT_TABLE table=default.Table_ee8a2c2391624131a4aac14a90311aa0 [id=c63ada5e02d74e049460990450ab4962]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.727006 979 tablet_replica.cc:331] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.727587 979 raft_consensus.cc:2241] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:31.727597 32346 tablet_service.cc:1468] Processing CreateTablet for tablet b0d1ec9d56ba40128325a50f204fe920 (DEFAULT_TABLE table=default.Table_3188d5c072e04adea5dae4c5ec7a71c8 [id=3e41f8259c5648c59b388ea9f4de99d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.728019 979 raft_consensus.cc:2270] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.728082 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (DEFAULT_TABLE table=default.Table_c76a280c52214058b9b4b2691090cc1f [id=ad7210d2665f463c8591e79cf62d15dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.728564 32360 tablet_service.cc:1468] Processing CreateTablet for tablet 45293db3c40b4e4194445f12e7475e66 (DEFAULT_TABLE table=default.Table_e8864c4ee64b4f4dbf04caa321024840 [id=537cece72a774ac99261a4c2a2e52c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.728603 979 ts_tablet_manager.cc:1905] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.729041 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 37b38cb54d6c45308358f556a692eabe (DEFAULT_TABLE table=default.Table_c4f705a7cbd940da941c460b61c5dece [id=5d3fcb492ac246419c43b0618b4d797d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.729552 32353 tablet_service.cc:1468] Processing CreateTablet for tablet a8de2b1105564fdab386442b1de4cedb (DEFAULT_TABLE table=default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.730034 32358 tablet_service.cc:1468] Processing CreateTablet for tablet f9f4829e103b460eb582beff3b966bb4 (DEFAULT_TABLE table=default.Table_c300ef24f71d4bebaa367db14adb1b8a [id=5242fe018e4a4a13a365961197786944]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.730518 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 306c4021b5d94a4996de07c222a8bfd8 (DEFAULT_TABLE table=default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.731151 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 9049e3d9b39f420bb51221808409fc54 (DEFAULT_TABLE table=default.Table_b7a92f8a00ba4931827eaee759bc8f91 [id=6fd0a78c429c40f8aa3b86e61e83a142]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.731694 32355 tablet_service.cc:1468] Processing CreateTablet for tablet f83f68ebda7e4826bbb17278cd4ac5a8 (DEFAULT_TABLE table=default.Table_6acb69304c424830a635f22f12343b31 [id=c1782341d0d34662889cff00d557ff81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.732280 32342 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:31.733227 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 789d1c20aa72480ab71123c799bbde4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.734484 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45293db3c40b4e4194445f12e7475e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.735464 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5af1dea3c6ca4f498ce6c039d72a14b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.737232 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b38cb54d6c45308358f556a692eabe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.741237 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9f4829e103b460eb582beff3b966bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.742122 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8de2b1105564fdab386442b1de4cedb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.746862 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d1ec9d56ba40128325a50f204fe920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.748028 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f83f68ebda7e4826bbb17278cd4ac5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.749923 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9049e3d9b39f420bb51221808409fc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.750988 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8a5e35c1974456b8079ab8907c2652e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.754904 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306c4021b5d94a4996de07c222a8bfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.709074 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:31.773005 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ee5a4fe03848229524cc0105226d8c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:31.773226 32654 raft_consensus.cc:2466] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:31.781344 979 ts_tablet_manager.cc:1918] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.783269 1011 raft_consensus.cc:491] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.785842 1011 raft_consensus.cc:513] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.786515 1011 leader_election.cc:290] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:31.786967 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 148564us
I20250623 08:04:31.787247 979 log.cc:1199] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/5629d11c44e241d598f3e519b2fe1a44
I20250623 08:04:31.787933 979 ts_tablet_manager.cc:1939] T 5629d11c44e241d598f3e519b2fe1a44 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.788633 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f9e0d1b85a94ae9ae86e53688a55db5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:31.788753 1011 raft_consensus.cc:491] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.788887 32654 raft_consensus.cc:2466] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:31.788915 1011 raft_consensus.cc:513] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.789201 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38ecd4c243b840f4b66d8e01078dca64" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:31.789408 32653 raft_consensus.cc:2466] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:31.789557 1011 leader_election.cc:290] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:31.790007 1021 consensus_queue.cc:1035] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:31.791446 657 consensus_queue.cc:1035] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:31.792420 32589 leader_election.cc:343] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.792595 32589 leader_election.cc:304] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.792886 1011 raft_consensus.cc:2747] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:31.793020 32589 leader_election.cc:304] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.793300 1011 raft_consensus.cc:2802] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:31.793409 32589 leader_election.cc:304] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.793439 1011 raft_consensus.cc:491] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:31.793541 1011 raft_consensus.cc:3058] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:31.793669 944 raft_consensus.cc:2802] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:31.793753 944 raft_consensus.cc:491] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:31.793819 32589 leader_election.cc:304] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.793869 944 raft_consensus.cc:3058] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:31.794729 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79731106915d4cebb9dcc3c10386ffa9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:31.795807 32589 leader_election.cc:343] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.796468 710 raft_consensus.cc:2802] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:31.796640 710 raft_consensus.cc:695] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:31.796902 944 raft_consensus.cc:513] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.797000 710 consensus_queue.cc:237] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.797506 944 leader_election.cc:290] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:31.798520 819 raft_consensus.cc:491] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.798693 819 raft_consensus.cc:513] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.799329 819 leader_election.cc:290] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:31.799563 1011 raft_consensus.cc:513] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.800221 1011 leader_election.cc:290] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:31.804762 979 tablet_replica.cc:331] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.805274 979 raft_consensus.cc:2241] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.805756 979 pending_rounds.cc:70] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:31.810982 979 pending_rounds.cc:77] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727400669351936 op_type: NO_OP noop_request { }
I20250623 08:04:31.811272 952 ts_tablet_manager.cc:1939] T c6a97b34e386443fa19a9601621b5017 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:31.811591 979 raft_consensus.cc:2887] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:31.811995 979 raft_consensus.cc:2270] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.813928 979 ts_tablet_manager.cc:1905] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.847136 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 771ecfc41e2b4007a4d213cec5b128c6 (DEFAULT_TABLE table=default.Table_bf77bbe8745846d3b962bd5b7fed73ce [id=2158e340258845d19f99a6300ec4c86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.847647 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 771ecfc41e2b4007a4d213cec5b128c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.852638 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 (DEFAULT_TABLE table=default.Table_04e85a0dbf974a05a94b9a9713fcb3f2 [id=7febcca3a06346738e869f73028a972e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:31.853078 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c9d1c8a67e149478a0d69842a4eb5b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:31.853140 979 ts_tablet_manager.cc:1918] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.853302 979 log.cc:1199] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/f3c850021e914530bb106ab7166be7c9
I20250623 08:04:31.853744 979 ts_tablet_manager.cc:1939] T f3c850021e914530bb106ab7166be7c9 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:31.856868 32453 leader_election.cc:343] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:31.857201 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 65727.6us
I20250623 08:04:31.857518 32453 leader_election.cc:304] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.858175 1021 raft_consensus.cc:2747] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:31.858783 819 raft_consensus.cc:491] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.858966 819 raft_consensus.cc:513] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:31.859771 32453 consensus_peers.cc:487] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:31.859858 819 leader_election.cc:290] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:31.859982 32453 leader_election.cc:304] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:31.858969 979 tablet_replica.cc:331] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.860265 699 raft_consensus.cc:2747] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:31.860572 979 raft_consensus.cc:2241] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.860821 979 raft_consensus.cc:2270] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.861455 979 ts_tablet_manager.cc:1905] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.861461 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7adec1482119405daca4aa603770ae22" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.862053 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79731106915d4cebb9dcc3c10386ffa9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:31.862391 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f44fd3c67185484e93450682364dc8a8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:31.862818 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38ecd4c243b840f4b66d8e01078dca64" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:31.863482 32349 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9e53da264a5ca1f70e1846c906f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:31.867449 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38ecd4c243b840f4b66d8e01078dca64" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:31.868290 32589 leader_election.cc:343] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.868574 979 ts_tablet_manager.cc:1918] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.868781 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea702abb31fb41498ab61bbb05b1f7e2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:31.871547 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48da1e4759bb4fea87a45ad8a428b6e7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:31.874485 746 raft_consensus.cc:491] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.878055 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:31.878185 979 log.cc:1199] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b980c746a3c042bfae90aa076acbb1df
I20250623 08:04:31.878722 979 ts_tablet_manager.cc:1939] T b980c746a3c042bfae90aa076acbb1df P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.874665 746 raft_consensus.cc:513] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:31.880652 1038 thread.cc:640] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.018s sys 0.013s
W20250623 08:04:31.880780 1038 thread.cc:606] raft [worker] (thread pool) Time spent starting thread: real 0.585s user 0.018s sys 0.013s
W20250623 08:04:31.871379 32589 leader_election.cc:343] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.870117 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:31.881239 32589 leader_election.cc:343] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.881312 32653 raft_consensus.cc:2466] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:31.882383 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f9e0d1b85a94ae9ae86e53688a55db5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:31.882419 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 43) took 1017 ms. Trace:
I20250623 08:04:31.882571 979 rpcz_store.cc:276] 0623 08:04:30.865277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.865321 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:31.858870 (+993549us) ts_tablet_manager.cc:1145] Deleting tablet b980c746a3c042bfae90aa076acbb1df
0623 08:04:31.858883 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:31.864905 (+ 6022us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:31.868395 (+ 3490us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:31.882365 (+ 13970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":993355}
W20250623 08:04:31.884111 32043 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 174244us
W20250623 08:04:31.884421 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet d6ed4cb252b1452087f8ee53ec78d985 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6ed4cb252b1452087f8ee53ec78d985 already in progress: creating tablet
I20250623 08:04:31.884830 979 tablet_replica.cc:331] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
W20250623 08:04:31.885907 32043 catalog_manager.cc:5007] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): alter failed for tablet 4ac9f38877b24d619c280a1aaa0b7b08 (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]): Aborted: Op aborted
I20250623 08:04:31.886323 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 7da84969a6464b5caeaf47af0cbaaa4d (table default.Table_fae6ac5a256346149e8cf56ae9dbdf3b [id=3c89160b4fb54f649fd3d222f6a6699d]) successfully deleted
I20250623 08:04:31.887815 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 5629d11c44e241d598f3e519b2fe1a44 (table default.Table_4d4535cab44545839eb3a559385e395a [id=bb849e86d7084959830c68207fb99061]) successfully deleted
W20250623 08:04:31.888080 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d6ed4cb252b1452087f8ee53ec78d985 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6ed4cb252b1452087f8ee53ec78d985 already in progress: creating tablet
W20250623 08:04:31.888101 32589 leader_election.cc:343] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.888348 979 raft_consensus.cc:2241] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.888576 979 raft_consensus.cc:2270] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:31.888883 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f4a0b9402a264800a1ea39707c10da0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4a0b9402a264800a1ea39707c10da0c already in progress: creating tablet
I20250623 08:04:31.889098 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet f3c850021e914530bb106ab7166be7c9 (table default.Table_fd49e04fcac14f0194cc9fed910601fc [id=17b055700da64cc1b19f93dd5edf40d5]) successfully deleted
I20250623 08:04:31.889777 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b980c746a3c042bfae90aa076acbb1df (table default.Table_71cce1d280bf41b7bae2f47aeec9a39e [id=09707632913546819a7f78d5d6bc5b2a]) successfully deleted
I20250623 08:04:31.890203 979 ts_tablet_manager.cc:1905] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.893049 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_98f40c99d9ff4819b958cd84cc5dc3a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.893744 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98f40c99d9ff4819b958cd84cc5dc3a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.895303 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_68acbd516e8f4990a2f197ac131bc6ba" } modify_external_catalogs: true
I20250623 08:04:31.895537 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_68acbd516e8f4990a2f197ac131bc6ba" } modify_external_catalogs: true
I20250623 08:04:31.901245 979 ts_tablet_manager.cc:1918] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:31.901484 32381 raft_consensus.cc:3058] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:31.908046 979 log.cc:1199] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/2ecf4af9e476495683bd63cac20d4672
I20250623 08:04:31.908824 979 ts_tablet_manager.cc:1939] T 2ecf4af9e476495683bd63cac20d4672 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:31.909862 32059 master_service.cc:948] ReplaceTablet: received request to replace tablet e9f8aa424bbb4aa8a907259270d46656 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:31.911056 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 44) took 1045 ms. Trace:
I20250623 08:04:31.911435 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet a9c8d480b8ba49a1be140b0e0a79f4e4
I20250623 08:04:31.911455 979 rpcz_store.cc:276] 0623 08:04:30.865842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.865887 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:31.884737 (+1018850us) ts_tablet_manager.cc:1145] Deleting tablet 2ecf4af9e476495683bd63cac20d4672
0623 08:04:31.884752 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:31.897812 (+ 13060us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:31.901134 (+ 3322us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:31.910830 (+ 9696us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018672}
I20250623 08:04:31.912201 32058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6889774a49f94dddaef8079edda86978" } new_table_name: "default.Table_4adeab2edc614899ad6cadb6d86fef97" modify_external_catalogs: true
I20250623 08:04:31.912710 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 2ecf4af9e476495683bd63cac20d4672 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:31.912811 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.913615 979 tablet_replica.cc:331] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.914320 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_5d6ec4f6805b4e0883b8d0419625d033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:31.914387 979 raft_consensus.cc:2241] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:31.915022 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d6ec4f6805b4e0883b8d0419625d033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.915166 979 raft_consensus.cc:2270] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.917136 979 ts_tablet_manager.cc:1905] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.918572 32057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_f5479d2204fd477ba893d16a0f77c4fa" } new_table_name: "default.Table_bc4ef67b97864fb99c405ffe92e2d006" modify_external_catalogs: true
W20250623 08:04:31.919281 32052 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet f3d32798168345edb9898751d083057c (table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f3d32798168345edb9898751d083057c consensus state has no leader
W20250623 08:04:31.919806 32052 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 38ecd4c243b840f4b66d8e01078dca64 (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 38ecd4c243b840f4b66d8e01078dca64 consensus state has no leader
I20250623 08:04:31.927163 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c850021e914530bb106ab7166be7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.927850 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e47b10f46b8402e94354ebca573c194 with delete_type TABLET_DATA_DELETED (replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.929243 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_5c43b8626a6640e4a61e495ab2ea5697"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.930017 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c43b8626a6640e4a61e495ab2ea5697 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.930864 32050 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_79be3d7247284828a82f02d96faa60cc" } modify_external_catalogs: true
I20250623 08:04:31.931090 32050 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_79be3d7247284828a82f02d96faa60cc" } modify_external_catalogs: true
I20250623 08:04:31.935079 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet 417a050eada4403ebbddff255fb6fb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.935298 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 175ee90133994154b7e4b58b0770db3b with delete_type TABLET_DATA_DELETED (replaced by tablet e602d391526d4b0eb0dbf5189afbe075) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.935539 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d with delete_type TABLET_DATA_DELETED (replaced by tablet 136e482995a24c5daec55a701f3091fa) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.935743 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 with delete_type TABLET_DATA_DELETED (replaced by tablet 205da880685b49b3b8669f73dc2c563b) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.935946 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 67c44697ecab4dd9b43f544a602897ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.936365 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 2060a643d0da47efa868f52bdb331b1f with delete_type TABLET_DATA_DELETED (replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.939406 819 raft_consensus.cc:491] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.939621 819 raft_consensus.cc:513] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.940418 819 leader_election.cc:290] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:31.943347 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.943588 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet b980c746a3c042bfae90aa076acbb1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:31.944294 32057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73453eb1a0394cfe8d49adcc71d4f3e9 consensus state has no leader
I20250623 08:04:31.944762 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_5cf87169ed0d4987b396195d518aa055"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:31.945381 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cf87169ed0d4987b396195d518aa055 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:31.947037 979 ts_tablet_manager.cc:1918] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:31.947585 979 log.cc:1199] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/228375592aee48a1aa21e1d16962b663
I20250623 08:04:31.947984 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet e9f8aa424bbb4aa8a907259270d46656
I20250623 08:04:31.948499 32059 catalog_manager.cc:6652] ReplaceTablet: tablet e9f8aa424bbb4aa8a907259270d46656 deleted and replaced by tablet a0e9b2f810014e81a29ca9b41cd3f8ba
I20250623 08:04:31.948822 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet e9f8aa424bbb4aa8a907259270d46656 with delete_type TABLET_DATA_DELETED (replaced by tablet a0e9b2f810014e81a29ca9b41cd3f8ba) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.948843 32059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_79f69fca0b25496e896d93fdd78e14b6" } modify_external_catalogs: true
I20250623 08:04:31.949041 32059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_79f69fca0b25496e896d93fdd78e14b6" } modify_external_catalogs: true
I20250623 08:04:31.949807 979 ts_tablet_manager.cc:1939] T 228375592aee48a1aa21e1d16962b663 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:31.950361 32058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74da35cd34c14284af004001b25aa531 has no consensus state
I20250623 08:04:31.950711 32058 master_service.cc:948] ReplaceTablet: received request to replace tablet 19536c16ff32460e9db571e4ce411bab from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:31.951359 32058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a65e08da5dbe4b408014e6c3b662450f" } modify_external_catalogs: true
I20250623 08:04:31.951577 32058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a65e08da5dbe4b408014e6c3b662450f" } modify_external_catalogs: true
I20250623 08:04:31.952966 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet ecb422550efd40cf82bc92fca5c25eda
I20250623 08:04:31.953720 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fcc07b378bf64122ae65bc17f130b2f0" } new_table_name: "default.Table_3c51bbf186704672adf120f825160733" modify_external_catalogs: true
I20250623 08:04:31.953929 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb422550efd40cf82bc92fca5c25eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:31.954107 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 42) took 1089 ms. Trace:
I20250623 08:04:31.954309 979 rpcz_store.cc:276] 0623 08:04:30.865051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.866079 (+ 1028us) service_pool.cc:225] Handling call
0623 08:04:31.913348 (+1047269us) ts_tablet_manager.cc:1145] Deleting tablet 228375592aee48a1aa21e1d16962b663
0623 08:04:31.913360 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:31.931179 (+ 17819us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:31.946656 (+ 15477us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:31.954076 (+ 7420us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047005}
I20250623 08:04:31.955170 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 228375592aee48a1aa21e1d16962b663 (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]) successfully deleted
I20250623 08:04:31.955505 979 tablet_replica.cc:331] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.956132 979 raft_consensus.cc:2241] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.956633 979 raft_consensus.cc:2270] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.958932 979 ts_tablet_manager.cc:1905] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.964500 32056 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a6ef1bf66cd24835896dabd3af080a6a" } new_table_name: "default.Table_33f948aa4f74464d86b927937ff981b9" modify_external_catalogs: true
W20250623 08:04:31.972362 32052 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f consensus state has no leader
W20250623 08:04:31.976696 32056 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c63029399994480ba7ccabcadc1ee5e0 consensus state has no leader
I20250623 08:04:31.982728 339 raft_consensus.cc:357] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:31.983045 979 ts_tablet_manager.cc:1918] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:31.983304 32381 raft_consensus.cc:1273] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Refusing update from remote peer 125745849172449ba4be8244d61fdcb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 08:04:31.983747 979 log.cc:1199] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/32c82141edea4068b54755a85c6d7fb8
I20250623 08:04:31.984102 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63029399994480ba7ccabcadc1ee5e0" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:31.984246 819 consensus_queue.cc:1035] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:31.986151 32589 leader_election.cc:343] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:31.987390 979 ts_tablet_manager.cc:1939] T 32c82141edea4068b54755a85c6d7fb8 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:31.989401 32054 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b84b8b8713774ab4896842feb9fb075e consensus state has no leader
I20250623 08:04:31.990048 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 41) took 1125 ms. Trace:
I20250623 08:04:31.990490 979 rpcz_store.cc:276] 0623 08:04:30.864809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.866817 (+ 2008us) service_pool.cc:225] Handling call
0623 08:04:31.955271 (+1088454us) ts_tablet_manager.cc:1145] Deleting tablet 32c82141edea4068b54755a85c6d7fb8
0623 08:04:31.955282 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:31.970770 (+ 15488us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:31.982526 (+ 11756us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:31.989916 (+ 7390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1088223}
I20250623 08:04:31.991945 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 32c82141edea4068b54755a85c6d7fb8 (table default.Table_c30b48097bd643b18b2a744839780b9e [id=7e59ad2235c1418d97f2b73ff2a48300]) successfully deleted
I20250623 08:04:31.992506 979 tablet_replica.cc:331] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:31.993058 819 raft_consensus.cc:491] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:31.993459 819 raft_consensus.cc:513] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:31.994103 819 leader_election.cc:290] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:31.994529 979 raft_consensus.cc:2241] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:31.995195 979 raft_consensus.cc:2270] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:31.997313 979 ts_tablet_manager.cc:1905] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:31.998678 339 raft_consensus.cc:383] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.003415 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 1a070895f4984493a3b9f6fa779797a9
I20250623 08:04:32.008440 32050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 3249605158634f4eab5b9d61357c383f
I20250623 08:04:32.009431 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet c7fcf583064f4538913d9157e93068dd
I20250623 08:04:32.013440 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3d32798168345edb9898751d083057c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:32.016309 32589 leader_election.cc:343] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.022667 979 ts_tablet_manager.cc:1918] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:32.023414 979 log.cc:1199] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/79731106915d4cebb9dcc3c10386ffa9
I20250623 08:04:32.024899 746 leader_election.cc:290] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:32.025146 32314 consensus_peers.cc:487] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 279624b8bb834b95a4f08045968ebfb4. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:32.025528 979 ts_tablet_manager.cc:1939] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:32.027782 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 40) took 1163 ms. Trace:
I20250623 08:04:32.028328 979 rpcz_store.cc:276] 0623 08:04:30.864538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.867078 (+ 2540us) service_pool.cc:225] Handling call
0623 08:04:31.992244 (+1125166us) ts_tablet_manager.cc:1145] Deleting tablet 79731106915d4cebb9dcc3c10386ffa9
0623 08:04:31.992257 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.008544 (+ 16287us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.022249 (+ 13705us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.027575 (+ 5326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124756}
W20250623 08:04:32.025415 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 145780us
I20250623 08:04:32.030294 979 tablet_replica.cc:331] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
W20250623 08:04:32.031041 32314 consensus_peers.cc:487] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0311ce67989f49e8866234a88e5b4d09. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:32.031371 979 raft_consensus.cc:2241] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.032075 979 raft_consensus.cc:2270] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.029837 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 79731106915d4cebb9dcc3c10386ffa9 (table default.Table_d44a2132435f43e7847bfa349645a605 [id=b42459e9ca5e475cb5083d68bb6349be]) successfully deleted
W20250623 08:04:32.032619 32314 consensus_peers.cc:487] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19536c16ff32460e9db571e4ce411bab. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:32.034526 979 ts_tablet_manager.cc:1905] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.045022 819 consensus_queue.cc:1035] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.052188 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 3c14e325fa5f4f848748c346bc25dde0 (DEFAULT_TABLE table=default.Table_f66b3fa5edc14cffaa95a0a0a4f62f67 [id=ad4cb54e99034dd3af8bdac8fcce4669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.052667 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c14e325fa5f4f848748c346bc25dde0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.055189 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 16bc969d09f74b0986070a504f9f670e (DEFAULT_TABLE table=default.Table_2891f87970d64c6383f346e5b51e9ad3 [id=e6e52d5a7c6c4bef978d652a1ca81e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.055595 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16bc969d09f74b0986070a504f9f670e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.058951 32517 raft_consensus.cc:3058] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:32.059079 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 2e898ede64e641519c8ee3ea1853a4b0 (DEFAULT_TABLE table=default.Table_303d632a02404fbe8b9baa353a8d3747 [id=670bededffc8499e8731b15f7e03380c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.059445 1054 raft_consensus.cc:2802] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:32.059689 32516 raft_consensus.cc:3058] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:32.059836 699 raft_consensus.cc:491] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.060094 952 tablet_replica.cc:331] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:32.060137 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea702abb31fb41498ab61bbb05b1f7e2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:32.060504 699 raft_consensus.cc:513] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.061147 699 leader_election.cc:290] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:32.062832 32632 tablet_service.cc:1468] Processing CreateTablet for tablet 1be8f8557b9a4ae2917c37b8d01cedf6 (DEFAULT_TABLE table=default.Table_e8b3bfa63ee7407ead7f4a0f8204d269 [id=02411177c0aa470b9e94c2714d561db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.063354 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e898ede64e641519c8ee3ea1853a4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.063359 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 6926c49b787941d98c647543bd3b9ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:32.062969 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:32.065902 1054 raft_consensus.cc:491] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250623 08:04:32.066468 32052 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.1:45613 (request call id 152) took 1837 ms (client timeout 1000 ms). Trace:
W20250623 08:04:32.066570 32052 rpcz_store.cc:269] 0623 08:04:30.228693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.272565 (+ 43872us) service_pool.cc:225] Handling call
0623 08:04:30.296913 (+ 24348us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f3b58 00007f19c7640176 00007f19c763fea5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:32.066402 (+1769489us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:30.780487 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:30.780635 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:30.780645 (+ 10us) write_op.cc:435] Acquired schema lock
0623 08:04:30.780649 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:30.780765 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:30.780780 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:30.780785 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:30.780889 (+ 104us) tablet.cc:699] Row locks acquired
0623 08:04:30.780891 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:30.780926 (+ 35us) write_op.cc:270] Start()
0623 08:04:30.780957 (+ 31us) write_op.cc:276] Timestamp: P: 1750665870780899 usec, L: 0
0623 08:04:30.780960 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:30.789064 (+ 8104us) log.cc:844] Serialized 901 byte log entry
0623 08:04:30.789257 (+ 193us) spinlock_profiling.cc:243] Waited 7.87 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 000000000097041a 00007f19c8ac25e2 00007f19c9e69255 00007f19c9e67d79 00007f19c9e701a0 00007f19c9e6ffad 0000000000a751bc 00007f19bd0d43ff 00007f19bd0d76cb 00007f19bd0d7548 0000000000a751bc
0623 08:04:30.794086 (+ 4829us) op_driver.cc:464] REPLICATION: finished
0623 08:04:30.794414 (+ 328us) write_op.cc:301] APPLY: starting
0623 08:04:30.794431 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:30.794475 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:30.794477 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:30.794547 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:30.794580 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:30.794587 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:30.794669 (+ 82us) log.cc:844] Serialized 29 byte log entry
0623 08:04:30.794730 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:30.794771 (+ 41us) write_op.cc:454] Released schema lock
0623 08:04:30.794778 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:30.794851 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:30.794855 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":102,"spinlock_wait_cycles":43739520,"child_traces":[["op",{"apply.queue_time_us":271,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":431,"num_ops":2,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":1201,"prepare.run_wall_time_us":9246,"raft.queue_time_us":423,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":300,"replication_time_us":13097,"spinlock_wait_cycles":18134784}]]}
I20250623 08:04:32.067502 952 raft_consensus.cc:2241] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.067517 32618 tablet_service.cc:1468] Processing CreateTablet for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (DEFAULT_TABLE table=default.Table_37287a2bae6a4d72ab9a957a92be554b [id=d5366c10bade4a2b93f0a2b2524517fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.068634 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f9e0d1b85a94ae9ae86e53688a55db5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:32.070808 342 tablet_bootstrap.cc:654] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.071924 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 8e17e1ca404b462da43aa1db7dc3b556 (DEFAULT_TABLE table=default.Table_6a8ea07f0b174864b81addf035ed1f34 [id=dfff0b62abf54558b0b04a3b5d8cd896]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.072065 32055 catalog_manager.cc:5582] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:32.072985 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48da1e4759bb4fea87a45ad8a428b6e7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:32.073630 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1be8f8557b9a4ae2917c37b8d01cedf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.073149 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 31a67b70cbc64bb3bfd5a34865109a01 (table default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e [id=7a569a9fe3bd4c608ddca48a734df1da]). Expected version 2 got 0
I20250623 08:04:32.076244 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet ba3d2fa01d574dd992dc904ac024ba3a (table default.Table_d496fd69b6f140b88b5abc86c8839ee2 [id=6c57db68f5ed45a3b64d1e3a02ce8a2a]). Expected version 1 got 0
I20250623 08:04:32.082556 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249605158634f4eab5b9d61357c383f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:32.082978 32624 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a070895f4984493a3b9f6fa779797a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:32.083534 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e17e1ca404b462da43aa1db7dc3b556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.091414 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7f3d2e67e84bfcbd45a21658f69e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.093731 1054 raft_consensus.cc:3058] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:32.094424 952 raft_consensus.cc:2270] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.095213 952 ts_tablet_manager.cc:1905] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.094581 32634 tablet_service.cc:1468] Processing CreateTablet for tablet a8945b6f49474648ac73bcb85285680a (DEFAULT_TABLE table=default.Table_ac62bae678de4b11a971f7bbccd50303 [id=4f4663a5cad14a3593e16a52ec8aeece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.096349 339 raft_consensus.cc:738] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.097414 32483 tablet_service.cc:1515] Processing DeleteTablet for tablet 52d0a356c74f4daea9093c998b4bf73e with delete_type TABLET_DATA_DELETED (replaced by tablet bb06a340f82047ac8ec3516edb08f4b6) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:32.098698 32631 tablet_service.cc:1468] Processing CreateTablet for tablet d6fdea59f71e4a9d9fca8ede4b9439d2 (DEFAULT_TABLE table=default.Table_b9220275c4064c11bb0618d6a459aab6 [id=392754cfbca449d887b33eb6ec5f0ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.099021 32623 tablet_service.cc:1515] Processing DeleteTablet for tablet c7fcf583064f4538913d9157e93068dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:32.099184 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8945b6f49474648ac73bcb85285680a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.100785 342 tablet_bootstrap.cc:492] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:32.101006 342 ts_tablet_manager.cc:1397] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.502s user 0.005s sys 0.003s
I20250623 08:04:32.103031 32058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_4ab4668aa0314e1eac46707fe99c1aad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:32.103487 32516 raft_consensus.cc:2466] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
W20250623 08:04:32.103665 32058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ab4668aa0314e1eac46707fe99c1aad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.104141 32520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63029399994480ba7ccabcadc1ee5e0" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:32.104430 32518 raft_consensus.cc:1273] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Refusing update from remote peer 125745849172449ba4be8244d61fdcb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250623 08:04:32.106004 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3d32798168345edb9898751d083057c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:32.107095 699 consensus_queue.cc:1035] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20250623 08:04:32.107753 657 consensus_queue.cc:1035] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20250623 08:04:32.111218 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fdea59f71e4a9d9fca8ede4b9439d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.111181 32624 tablet_service.cc:1468] Processing CreateTablet for tablet 3740dd120745493688eeb4c1ac311dcb (DEFAULT_TABLE table=default.Table_7202dd9a46994e949c2368ac8f51b53d [id=4d97ab7c1137414ca359d29f66e8497b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.111224 1058 tablet.cc:1722] T 2414d24ab6c54f36bb1e554f7231a5d7 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:32.111618 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3740dd120745493688eeb4c1ac311dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.111815 32513 raft_consensus.cc:3058] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:32.112571 1071 tablet.cc:1722] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:32.112586 342 raft_consensus.cc:357] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.113380 342 raft_consensus.cc:383] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.113505 342 raft_consensus.cc:738] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.113761 342 consensus_queue.cc:260] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.114442 342 ts_tablet_manager.cc:1428] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250623 08:04:32.118422 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_28d324870b1049289c32fbfcf8c44ebf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.120838 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28d324870b1049289c32fbfcf8c44ebf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.132633 339 consensus_queue.cc:260] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:32.136313 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_6b3f6aad5faf4f2a88573140691c6291"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.136902 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b3f6aad5faf4f2a88573140691c6291 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.133428 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7adec1482119405daca4aa603770ae22" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:32.138597 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_90920cfec34b4621bae5567c93e73c89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:32.138792 819 consensus_queue.cc:1035] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:32.139122 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90920cfec34b4621bae5567c93e73c89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.142302 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_ff6e3cec929a433b8ac68152b9095dfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.142652 32314 consensus_peers.cc:487] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 417a050eada4403ebbddff255fb6fb5a. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:32.142915 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff6e3cec929a433b8ac68152b9095dfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.145104 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_2891f87970d64c6383f346e5b51e9ad3" } new_table_name: "default.Table_bc1251ce708a4467b5b1c71a1df307e0" modify_external_catalogs: true
I20250623 08:04:32.146831 32059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_91d48bbc63414bb4a1fb90f438f52eba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.147302 32059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91d48bbc63414bb4a1fb90f438f52eba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.148947 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_bf77bbe8745846d3b962bd5b7fed73ce" } new_table_name: "default.Table_41ce11c5261f4bcfb883fadb408fddc0" modify_external_catalogs: true
I20250623 08:04:32.150295 32058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d226f9dab24f45ae896579983b3d1a67" } modify_external_catalogs: true
I20250623 08:04:32.150527 32058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d226f9dab24f45ae896579983b3d1a67" } modify_external_catalogs: true
I20250623 08:04:32.153506 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a615b39019e1478aa69bd46fc2e4a4a9" } new_table_name: "default.Table_d94ea6a713d546fa80c002017bd9a39f" modify_external_catalogs: true
W20250623 08:04:32.164098 32589 leader_election.cc:343] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.162919 32632 tablet_service.cc:1468] Processing CreateTablet for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7 (DEFAULT_TABLE table=default.Table_35a9f061c58a4dd1b8ad6e194d32bbde [id=764330c0798042d3a0ca1e6d0f1ec78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.166481 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:32.169996 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 79167us
I20250623 08:04:32.173394 32513 raft_consensus.cc:2466] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:32.174238 342 tablet_bootstrap.cc:492] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:32.174306 1058 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:55526 (request call id 51) took 1207 ms. Trace:
I20250623 08:04:32.174440 1058 rpcz_store.cc:276] 0623 08:04:30.966401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.966692 (+ 291us) service_pool.cc:225] Handling call
0623 08:04:32.174260 (+1207568us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:31.153081 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:31.181318 (+ 28237us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:31.181327 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:31.181334 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:31.181383 (+ 49us) alter_schema_op.cc:128] START. Timestamp: 1750665871181349
0623 08:04:31.181386 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:31.181639 (+ 253us) log.cc:844] Serialized 173 byte log entry
0623 08:04:32.085186 (+903547us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.085639 (+ 453us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:32.111623 (+ 25984us) log.cc:844] Serialized 5 byte log entry
0623 08:04:32.173632 (+ 62009us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.173973 (+ 341us) log.cc:844] Serialized 14 byte log entry
0623 08:04:32.174213 (+ 240us) alter_schema_op.cc:73] Released schema lock
0623 08:04:32.174221 (+ 8us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":393,"prepare.queue_time_us":172962,"prepare.run_cpu_time_us":4793,"prepare.run_wall_time_us":272304,"raft.queue_time_us":81,"raft.run_cpu_time_us":656,"raft.run_wall_time_us":655,"replication_time_us":903772,"thread_start_us":335973,"threads_started":2,"wal-append.queue_time_us":241641}]]}
I20250623 08:04:32.178104 1071 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:55526 (request call id 30) took 1224 ms. Trace:
I20250623 08:04:32.178606 342 tablet_bootstrap.cc:654] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.178236 1071 rpcz_store.cc:276] 0623 08:04:30.954051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.954094 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:32.178057 (+1223963us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:30.954522 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:30.954632 (+ 110us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:30.954652 (+ 20us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:30.954660 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:30.954697 (+ 37us) alter_schema_op.cc:128] START. Timestamp: 1750665870954634
0623 08:04:30.954700 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:30.954913 (+ 213us) log.cc:844] Serialized 173 byte log entry
0623 08:04:32.084869 (+1129956us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.085016 (+ 147us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:32.113050 (+ 28034us) log.cc:844] Serialized 5 byte log entry
0623 08:04:32.177710 (+ 64660us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.177970 (+ 260us) log.cc:844] Serialized 14 byte log entry
0623 08:04:32.178010 (+ 40us) alter_schema_op.cc:73] Released schema lock
0623 08:04:32.178017 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":87,"mutex_wait_us":21,"prepare.queue_time_us":34,"prepare.run_cpu_time_us":10619,"prepare.run_wall_time_us":223046,"raft.queue_time_us":69,"raft.run_cpu_time_us":501,"raft.run_wall_time_us":501,"replication_time_us":1130126,"thread_start_us":221326,"threads_started":1,"wal-append.queue_time_us":195213}]]}
I20250623 08:04:32.181208 1054 raft_consensus.cc:513] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.181864 1054 leader_election.cc:290] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:32.184612 32517 raft_consensus.cc:2466] T f44fd3c67185484e93450682364dc8a8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:32.185278 32514 raft_consensus.cc:2391] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:32.187858 339 ts_tablet_manager.cc:1428] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.433s user 0.003s sys 0.000s
I20250623 08:04:32.188778 1001 consensus_queue.cc:1035] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250623 08:04:32.189188 339 tablet_bootstrap.cc:492] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:32.189484 342 tablet_bootstrap.cc:492] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:32.189668 342 ts_tablet_manager.cc:1397] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250623 08:04:32.191550 342 raft_consensus.cc:357] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.191783 342 raft_consensus.cc:383] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.191851 342 raft_consensus.cc:738] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.192168 32314 leader_election.cc:304] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:32.192608 1001 raft_consensus.cc:2802] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:32.192718 1001 raft_consensus.cc:491] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:32.192790 1001 raft_consensus.cc:3058] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:32.193048 342 consensus_queue.cc:260] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:32.193575 32314 leader_election.cc:343] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: ce1f3ef3690c4837b95b105733884f1f
I20250623 08:04:32.193749 342 ts_tablet_manager.cc:1428] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:32.194115 342 tablet_bootstrap.cc:492] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:32.194289 746 consensus_queue.cc:1035] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.195328 32314 leader_election.cc:304] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:32.195647 746 raft_consensus.cc:2802] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:32.195725 746 raft_consensus.cc:491] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:32.195813 746 raft_consensus.cc:3058] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:32.196053 342 tablet_bootstrap.cc:654] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.196681 1065 consensus_queue.cc:1035] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.199833 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_6ca32a3252134e6587bca37cff71c969"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.200515 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ca32a3252134e6587bca37cff71c969 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:32.203811 32050 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16bc969d09f74b0986070a504f9f670e consensus state has no leader
I20250623 08:04:32.205192 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_4d6c00c8083e4821aa4a209e9cee8a91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.205803 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d6c00c8083e4821aa4a209e9cee8a91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.207131 32059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_2fceb5254f914cd9a952ae0a805150c5" } new_table_name: "default.Table_19247d878b0b463aaadf4577f627c2ad" modify_external_catalogs: true
I20250623 08:04:32.208684 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_969668212d4f422ca8f2f34266315161"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.209304 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_969668212d4f422ca8f2f34266315161 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.211458 32057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_06b64498b6e148d5b86187f2ce7e4c31" } modify_external_catalogs: true
I20250623 08:04:32.211650 32057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_06b64498b6e148d5b86187f2ce7e4c31" } modify_external_catalogs: true
I20250623 08:04:32.211966 979 ts_tablet_manager.cc:1918] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:32.212553 979 log.cc:1199] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/2795cb14e6db4116a4c9ee539bd4e82d
I20250623 08:04:32.213227 1011 raft_consensus.cc:491] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.213404 848 raft_consensus.cc:491] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.213364 1011 raft_consensus.cc:513] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.213995 1011 leader_election.cc:290] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:32.215950 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 771ecfc41e2b4007a4d213cec5b128c6 consensus state has no leader
I20250623 08:04:32.221030 979 ts_tablet_manager.cc:1939] T 2795cb14e6db4116a4c9ee539bd4e82d P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:32.225111 848 raft_consensus.cc:513] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.228436 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 39) took 1363 ms. Trace:
I20250623 08:04:32.229151 979 rpcz_store.cc:276] 0623 08:04:30.864291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.867428 (+ 3137us) service_pool.cc:225] Handling call
0623 08:04:32.030013 (+1162585us) ts_tablet_manager.cc:1145] Deleting tablet 2795cb14e6db4116a4c9ee539bd4e82d
0623 08:04:32.030030 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.182131 (+152101us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.211369 (+ 29238us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.227950 (+ 16581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162138}
I20250623 08:04:32.231578 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 2795cb14e6db4116a4c9ee539bd4e82d (table default.Table_cd24de3acd89483480238a058a33e72c [id=9b918ea1a6ca412d9db93bbb1215732f]) successfully deleted
W20250623 08:04:32.231814 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 112980us
I20250623 08:04:32.232636 952 ts_tablet_manager.cc:1918] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:32.233517 979 tablet_replica.cc:331] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:32.233721 848 leader_election.cc:290] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:32.234673 952 log.cc:1199] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/2ecf4af9e476495683bd63cac20d4672
I20250623 08:04:32.235702 342 tablet_bootstrap.cc:492] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:32.235854 342 ts_tablet_manager.cc:1397] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.002s
I20250623 08:04:32.236023 952 ts_tablet_manager.cc:1939] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:32.235591 979 raft_consensus.cc:2241] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.236487 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 33) took 1281 ms. Trace:
I20250623 08:04:32.236763 342 raft_consensus.cc:357] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.236964 342 raft_consensus.cc:383] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.237067 342 raft_consensus.cc:738] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.237341 342 consensus_queue.cc:260] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.237287 979 raft_consensus.cc:2270] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.237931 342 ts_tablet_manager.cc:1428] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:32.238267 342 tablet_bootstrap.cc:492] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:32.240183 342 tablet_bootstrap.cc:654] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.240584 979 ts_tablet_manager.cc:1905] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.243299 1060 consensus_queue.cc:1035] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20250623 08:04:32.244685 32314 consensus_peers.cc:487] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0311ce67989f49e8866234a88e5b4d09. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:32.236685 952 rpcz_store.cc:276] 0623 08:04:30.955479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.955525 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:31.853107 (+897582us) ts_tablet_manager.cc:1145] Deleting tablet 2ecf4af9e476495683bd63cac20d4672
0623 08:04:31.853116 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.108596 (+255480us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.196533 (+ 87937us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.236440 (+ 39907us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2227,"tablet-delete.queue_time_us":897353}
I20250623 08:04:32.252142 952 tablet_replica.cc:331] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:32.252460 952 raft_consensus.cc:2241] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.252713 952 raft_consensus.cc:2270] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.253082 32453 leader_election.cc:304] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:32.253400 848 raft_consensus.cc:2762] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250623 08:04:32.253481 1001 raft_consensus.cc:513] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.253710 32453 leader_election.cc:304] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:32.253976 848 raft_consensus.cc:2762] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250623 08:04:32.254204 1001 leader_election.cc:290] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:32.262710 952 ts_tablet_manager.cc:1905] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.267081 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet c6a97b34e386443fa19a9601621b5017 (table default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca [id=c330ef9f7f0549a7abc1eeab676ec11d]) successfully deleted
W20250623 08:04:32.269479 32589 leader_election.cc:343] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.269743 32589 leader_election.cc:304] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.270102 1011 raft_consensus.cc:2747] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:32.270102 343 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.056s sys 0.146s
W20250623 08:04:32.270653 343 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.551s user 0.056s sys 0.146s
W20250623 08:04:32.271011 32589 leader_election.cc:343] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.271083 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:32.271282 32589 leader_election.cc:304] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:32.271586 32589 raft_consensus.cc:2684] T 79731106915d4cebb9dcc3c10386ffa9 P 125745849172449ba4be8244d61fdcb3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:32.271648 32055 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:32.278568 343 tablet_bootstrap.cc:492] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:32.278914 343 ts_tablet_manager.cc:1397] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.564s user 0.057s sys 0.148s
I20250623 08:04:32.280232 343 raft_consensus.cc:357] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.280629 343 raft_consensus.cc:383] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.280551 979 ts_tablet_manager.cc:1918] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:32.281176 343 raft_consensus.cc:738] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.281477 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31a67b70cbc64bb3bfd5a34865109a01" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:32.281769 32589 leader_election.cc:343] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.281730 343 consensus_queue.cc:260] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.281541 979 log.cc:1199] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/726d80e05a344f27b5a289723cb5aed7
I20250623 08:04:32.282181 32589 leader_election.cc:304] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.282709 1011 raft_consensus.cc:2747] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:32.283365 343 ts_tablet_manager.cc:1428] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250623 08:04:32.282667 32380 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.004s sys 0.008s
I20250623 08:04:32.284521 981 ts_tablet_manager.cc:1918] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:32.287338 746 raft_consensus.cc:513] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.287667 343 tablet_bootstrap.cc:492] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:32.291066 979 ts_tablet_manager.cc:1939] T 726d80e05a344f27b5a289723cb5aed7 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:32.291399 343 tablet_bootstrap.cc:654] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:32.293697 32353 consensus_meta.cc:287] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Time spent flushing consensus metadata: real 0.510s user 0.004s sys 0.003s
W20250623 08:04:32.295111 32380 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s user 0.004s sys 0.008s
I20250623 08:04:32.297211 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 45) took 1423 ms. Trace:
I20250623 08:04:32.299093 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0773fc60d4a463596f536f15d58bf57" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:32.298135 979 rpcz_store.cc:276] 0623 08:04:30.874069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.874115 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:32.232463 (+1358348us) ts_tablet_manager.cc:1145] Deleting tablet 726d80e05a344f27b5a289723cb5aed7
0623 08:04:32.232702 (+ 239us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.259971 (+ 27269us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.279760 (+ 19789us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.297009 (+ 17249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1358062}
W20250623 08:04:32.299530 32589 leader_election.cc:343] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.299695 32589 leader_election.cc:304] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.300040 944 raft_consensus.cc:2747] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:32.299806 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 726d80e05a344f27b5a289723cb5aed7 (table default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]) successfully deleted
I20250623 08:04:32.300480 1097 tablet.cc:1722] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:32.301625 746 leader_election.cc:290] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:32.301905 339 tablet_bootstrap.cc:654] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.301597 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 46) took 1427 ms. Trace:
I20250623 08:04:32.302471 979 rpcz_store.cc:276] 0623 08:04:30.874357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.874415 (+ 58us) service_pool.cc:225] Handling call
0623 08:04:32.300972 (+1426557us) ts_tablet_manager.cc:1145] Deleting tablet c0f87fd62f1545089d00757f8214593a
0623 08:04:32.300987 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.301389 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426176}
W20250623 08:04:32.303998 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet c0f87fd62f1545089d00757f8214593a because the tablet was not found. No further retry: Not found: Tablet not found: c0f87fd62f1545089d00757f8214593a
I20250623 08:04:32.305089 979 tablet_replica.cc:331] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:32.307075 981 log.cc:1199] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/4ac9f38877b24d619c280a1aaa0b7b08
I20250623 08:04:32.307030 979 raft_consensus.cc:2241] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.307829 981 ts_tablet_manager.cc:1939] T 4ac9f38877b24d619c280a1aaa0b7b08 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:32.307220 1113 tablet.cc:1722] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:32.308293 979 raft_consensus.cc:2270] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.308728 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 22) took 1686 ms. Trace:
I20250623 08:04:32.308914 981 rpcz_store.cc:276] 0623 08:04:30.622137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.622930 (+ 793us) service_pool.cc:225] Handling call
0623 08:04:31.214977 (+592047us) ts_tablet_manager.cc:1145] Deleting tablet 4ac9f38877b24d619c280a1aaa0b7b08
0623 08:04:31.214986 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:31.274146 (+ 59160us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x616000140e54. stack: 00007f7f6e5c4d56 00007f7f6e5c4b10 00007f7f6d5bb29d 0000000000970a84 0000000000970a4d 000000000097041a 00007f7f7a022e80 00007f7f7b334a93 00007f7f7de10c97 00007f7f7de42d24 00007f7f7de24780 00007f7f7de2450d 0000000000a751bc 00007f7f6e60a3ff 00007f7f6e60d6cb 00007f7f6e60d548
0623 08:04:31.634094 (+359948us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.258960 (+624866us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.308670 (+ 49710us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21250,"spinlock_wait_cycles":86381824,"tablet-delete.queue_time_us":585472}
I20250623 08:04:32.309943 981 tablet_replica.cc:331] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:32.310222 981 raft_consensus.cc:2241] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:32.310640 981 pending_rounds.cc:70] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:32.312722 979 ts_tablet_manager.cc:1905] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:32.326738 32589 leader_election.cc:343] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.332808 944 consensus_queue.cc:1035] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.333546 32589 leader_election.cc:304] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.333860 944 raft_consensus.cc:2802] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:32.334022 944 raft_consensus.cc:695] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:32.334398 944 consensus_queue.cc:237] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:32.342615 32589 leader_election.cc:343] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.343779 32589 leader_election.cc:304] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.344646 819 raft_consensus.cc:2747] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:32.344867 819 consensus_queue.cc:1035] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.343010 1121 tablet.cc:1722] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:32.344873 32589 leader_election.cc:343] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.349262 32589 leader_election.cc:304] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.349617 710 raft_consensus.cc:2747] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:32.350461 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:32.350729 32517 raft_consensus.cc:2466] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:32.363818 981 pending_rounds.cc:77] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7170727407769006080 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "0311ce67989f49e8866234a88e5b4d09" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_e962b9945a5041039f7e33b98b653175" dest_uuid: "7351aa31800245659edc9837e27daffd" new_extra_config { } }
W20250623 08:04:32.364333 981 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35542: Aborted: Op aborted
I20250623 08:04:32.364584 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:35542 (request call id 30) took 1367 ms. Trace:
W20250623 08:04:32.319563 32375 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.005s sys 0.009s
W20250623 08:04:32.365232 32375 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s user 0.005s sys 0.009s
W20250623 08:04:32.319689 32378 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.013s sys 0.000s
W20250623 08:04:32.365677 32378 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.013s sys 0.000s
I20250623 08:04:32.361793 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.367308 1118 tablet.cc:1722] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:32.373737 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 2ecf4af9e476495683bd63cac20d4672 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:32.376328 339 tablet_bootstrap.cc:492] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:32.376571 339 ts_tablet_manager.cc:1397] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.188s user 0.003s sys 0.006s
I20250623 08:04:32.377472 339 raft_consensus.cc:357] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.377863 339 raft_consensus.cc:383] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.377986 339 raft_consensus.cc:738] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.379187 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.380645 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 2060a643d0da47efa868f52bdb331b1f with delete_type TABLET_DATA_DELETED (replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.380988 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ecf4af9e476495683bd63cac20d4672 with delete_type TABLET_DATA_DELETED (replaced by tablet 52d0a356c74f4daea9093c998b4bf73e) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.381850 1097 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:35542 (request call id 31) took 1375 ms. Trace:
I20250623 08:04:32.364771 981 rpcz_store.cc:276] 0623 08:04:30.997600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.997678 (+ 78us) service_pool.cc:225] Handling call
0623 08:04:32.364554 (+1366876us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:31.035007 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:31.037366 (+ 2359us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:31.037376 (+ 10us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:31.037383 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:31.037430 (+ 47us) alter_schema_op.cc:128] START. Timestamp: 1750665871037355
0623 08:04:31.037435 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:31.037713 (+ 278us) log.cc:844] Serialized 173 byte log entry
0623 08:04:32.364263 (+1326550us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.364276 (+ 13us) op_driver.cc:389] HandleFailure(Aborted: Op aborted)
0623 08:04:32.364281 (+ 5us) alter_schema_op.cc:171] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"mutex_wait_us":96094,"prepare.queue_time_us":36857,"prepare.run_cpu_time_us":12185,"prepare.run_wall_time_us":148663,"raft.queue_time_us":97279,"raft.run_cpu_time_us":487,"raft.run_wall_time_us":8380,"replication_time_us":1326773,"thread_start_us":89790,"threads_started":2,"wal-append.queue_time_us":50083,"wal-append.run_cpu_time_us":585,"wal-append.run_wall_time_us":1000631}]]}
I20250623 08:04:32.384935 981 raft_consensus.cc:2270] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.388527 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31a67b70cbc64bb3bfd5a34865109a01" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:32.388767 32513 raft_consensus.cc:2391] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:32.390542 1071 tablet.cc:1722] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:32.391280 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 279624b8bb834b95a4f08045968ebfb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.391539 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.392115 1123 tablet.cc:1722] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:32.397047 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e47b10f46b8402e94354ebca573c194 with delete_type TABLET_DATA_DELETED (replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.397343 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 19536c16ff32460e9db571e4ce411bab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
W20250623 08:04:32.397626 32044 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 771ecfc41e2b4007a4d213cec5b128c6 (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 771ecfc41e2b4007a4d213cec5b128c6 consensus state has no leader
I20250623 08:04:32.399300 981 ts_tablet_manager.cc:1905] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.382025 1097 rpcz_store.cc:276] 0623 08:04:31.006269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.006333 (+ 64us) service_pool.cc:225] Handling call
0623 08:04:32.381816 (+1375483us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:31.183712 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:31.183829 (+ 117us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:31.183837 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:31.183844 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:31.183881 (+ 37us) alter_schema_op.cc:128] START. Timestamp: 1750665871183836
0623 08:04:31.183884 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:31.184057 (+ 173us) log.cc:844] Serialized 173 byte log entry
0623 08:04:32.062476 (+878419us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.300420 (+237944us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:32.301136 (+ 716us) log.cc:844] Serialized 5 byte log entry
0623 08:04:32.317894 (+ 16758us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.318184 (+ 290us) log.cc:844] Serialized 14 byte log entry
0623 08:04:32.381772 (+ 63588us) alter_schema_op.cc:73] Released schema lock
0623 08:04:32.381777 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":237869,"mutex_wait_us":12481,"prepare.queue_time_us":176831,"prepare.run_cpu_time_us":41484,"prepare.run_wall_time_us":697160,"raft.queue_time_us":17748,"raft.run_cpu_time_us":468,"raft.run_wall_time_us":468,"replication_time_us":878556,"thread_start_us":1206828,"threads_started":4,"wal-append.queue_time_us":98501}]]}
I20250623 08:04:32.410740 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 with delete_type TABLET_DATA_DELETED (replaced by tablet 205da880685b49b3b8669f73dc2c563b) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.411126 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 175ee90133994154b7e4b58b0770db3b with delete_type TABLET_DATA_DELETED (replaced by tablet e602d391526d4b0eb0dbf5189afbe075) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.411734 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.412057 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet d4221d7500d1458db75926d7b274a654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.412386 339 consensus_queue.cc:260] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.413300 339 ts_tablet_manager.cc:1428] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.037s user 0.000s sys 0.002s
I20250623 08:04:32.413903 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 32c82141edea4068b54755a85c6d7fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 162308da898d4dde8ab5b1ab01d0f393) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.414193 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.414855 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet b980c746a3c042bfae90aa076acbb1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.415172 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 228375592aee48a1aa21e1d16962b663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.415805 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 with delete_type TABLET_DATA_DELETED (replaced by tablet 31a67b70cbc64bb3bfd5a34865109a01) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.416085 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet d0ee5a4fe03848229524cc0105226d8c with delete_type TABLET_DATA_DELETED (replaced by tablet 38ecd4c243b840f4b66d8e01078dca64) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.419538 848 consensus_queue.cc:1035] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.419984 1113 tablet.cc:1722] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250623 08:04:32.421420 32453 leader_election.cc:343] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.421568 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.421905 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 205da880685b49b3b8669f73dc2c563b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.422143 32483 tablet_service.cc:1515] Processing DeleteTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:32.423095 32483 tablet_service.cc:1515] Processing DeleteTablet for tablet e9f8aa424bbb4aa8a907259270d46656 with delete_type TABLET_DATA_DELETED (replaced by tablet a0e9b2f810014e81a29ca9b41cd3f8ba) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:32.423529 32483 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb422550efd40cf82bc92fca5c25eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:32.424619 32483 tablet_service.cc:1468] Processing CreateTablet for tablet 3c14e325fa5f4f848748c346bc25dde0 (DEFAULT_TABLE table=default.Table_f66b3fa5edc14cffaa95a0a0a4f62f67 [id=ad4cb54e99034dd3af8bdac8fcce4669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.425052 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c14e325fa5f4f848748c346bc25dde0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.425283 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet e9f8aa424bbb4aa8a907259270d46656 with delete_type TABLET_DATA_DELETED (replaced by tablet a0e9b2f810014e81a29ca9b41cd3f8ba) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.425619 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dc696236a15420e9666cca708068fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.427456 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb422550efd40cf82bc92fca5c25eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.427832 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet a414144276f6484ea4690204752e2142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.428100 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 67c44697ecab4dd9b43f544a602897ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.428383 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe30079045d437aa61bb5bfbfb96133 with delete_type TABLET_DATA_DELETED (replaced by tablet 18ea63263b9b466e922169eecafab747) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.428714 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 162308da898d4dde8ab5b1ab01d0f393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.429061 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 726d80e05a344f27b5a289723cb5aed7 with delete_type TABLET_DATA_DELETED (replaced by tablet 432d1642c0e44119b5bc05858463a136) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.429355 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f87fd62f1545089d00757f8214593a with delete_type TABLET_DATA_DELETED (replaced by tablet 1f76ef24c69f4b17a373b242d71da8b9) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.429667 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet ce1f3ef3690c4837b95b105733884f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.429977 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ea81642050141a9be84f151a5785abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.430450 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.430809 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f with delete_type TABLET_DATA_DELETED (replaced by tablet b1aa596950da49a898e3ed9bbad8c45e) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.431365 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f9e0d1b85a94ae9ae86e53688a55db5) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.431586 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 776b412b3f564a7581c6a5e1c84e8499 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.431919 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.432379 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet cde7df7d39a54114a4dcb28cfd1eb429 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f1fb8231e8e45208ac4f412134d8696) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.432575 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet c6a97b34e386443fa19a9601621b5017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.438235 339 tablet_bootstrap.cc:492] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:32.440826 339 tablet_bootstrap.cc:654] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.448633 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet b80a91ae2c084020b5d3444b570b28f2
W20250623 08:04:32.449344 32054 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7adec1482119405daca4aa603770ae22 consensus state has no leader
I20250623 08:04:32.449708 32054 master_service.cc:948] ReplaceTablet: received request to replace tablet 9918b15e48144ef0b8f3aedc2135be59 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:32.450220 32054 master_service.cc:948] ReplaceTablet: received request to replace tablet 38ecd4c243b840f4b66d8e01078dca64 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:32.450683 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_a0e4ca05337b448ca84bafd20b982a81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.451306 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0e4ca05337b448ca84bafd20b982a81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.452620 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_bc4ef67b97864fb99c405ffe92e2d006" } new_table_name: "default.Table_45c6b0778f6d4874afc4875669e6b179" modify_external_catalogs: true
I20250623 08:04:32.452713 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet b80a91ae2c084020b5d3444b570b28f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:32.453848 343 tablet_bootstrap.cc:492] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:32.454125 343 ts_tablet_manager.cc:1397] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.167s user 0.016s sys 0.017s
I20250623 08:04:32.457578 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet ae7c2ac546114a819a543f1667fb7844 with delete_type TABLET_DATA_DELETED (replaced by tablet d98654c29b9c4e75af6a213cafd4c2c9) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.458020 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 966a159e212040109e2842c15b69fb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.458711 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 3c14e325fa5f4f848748c346bc25dde0 (DEFAULT_TABLE table=default.Table_f66b3fa5edc14cffaa95a0a0a4f62f67 [id=ad4cb54e99034dd3af8bdac8fcce4669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.459225 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c14e325fa5f4f848748c346bc25dde0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.459313 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet c7fcf583064f4538913d9157e93068dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:32.459949 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 16bc969d09f74b0986070a504f9f670e (DEFAULT_TABLE table=default.Table_2891f87970d64c6383f346e5b51e9ad3 [id=e6e52d5a7c6c4bef978d652a1ca81e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.460316 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16bc969d09f74b0986070a504f9f670e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.462993 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a070895f4984493a3b9f6fa779797a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.463476 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249605158634f4eab5b9d61357c383f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.464193 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet c7fcf583064f4538913d9157e93068dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.465720 32652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:32.466043 32652 raft_consensus.cc:2391] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:32.466626 32483 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a070895f4984493a3b9f6fa779797a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:32.466779 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d with delete_type TABLET_DATA_DELETED (replaced by tablet 136e482995a24c5daec55a701f3091fa) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.467126 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet f0117cbb09dd4067836266c9d6a32adf with delete_type TABLET_DATA_DELETED (replaced by tablet 2ecf4af9e476495683bd63cac20d4672) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.470562 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 417a050eada4403ebbddff255fb6fb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.471236 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 16bc969d09f74b0986070a504f9f670e (DEFAULT_TABLE table=default.Table_2891f87970d64c6383f346e5b51e9ad3 [id=e6e52d5a7c6c4bef978d652a1ca81e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.471712 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16bc969d09f74b0986070a504f9f670e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.473600 952 ts_tablet_manager.cc:1918] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:32.473824 952 log.cc:1199] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/ea702abb31fb41498ab61bbb05b1f7e2
I20250623 08:04:32.474272 952 ts_tablet_manager.cc:1939] T ea702abb31fb41498ab61bbb05b1f7e2 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:32.476945 32652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:32.477144 32652 raft_consensus.cc:2466] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:32.478128 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 2e898ede64e641519c8ee3ea1853a4b0 (DEFAULT_TABLE table=default.Table_303d632a02404fbe8b9baa353a8d3747 [id=670bededffc8499e8731b15f7e03380c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.478533 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e898ede64e641519c8ee3ea1853a4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.478885 1119 tablet.cc:1722] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:32.481400 32053 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_48cbdc58fd034e6390087ed7c6346d24" } modify_external_catalogs: true
I20250623 08:04:32.481571 32053 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_48cbdc58fd034e6390087ed7c6346d24" } modify_external_catalogs: true
I20250623 08:04:32.484004 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249605158634f4eab5b9d61357c383f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:32.484653 32483 tablet_service.cc:1468] Processing CreateTablet for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (DEFAULT_TABLE table=default.Table_37287a2bae6a4d72ab9a957a92be554b [id=d5366c10bade4a2b93f0a2b2524517fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.485101 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7f3d2e67e84bfcbd45a21658f69e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.487030 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 2e898ede64e641519c8ee3ea1853a4b0 (DEFAULT_TABLE table=default.Table_303d632a02404fbe8b9baa353a8d3747 [id=670bededffc8499e8731b15f7e03380c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.487459 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e898ede64e641519c8ee3ea1853a4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.488314 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 1be8f8557b9a4ae2917c37b8d01cedf6 (DEFAULT_TABLE table=default.Table_e8b3bfa63ee7407ead7f4a0f8204d269 [id=02411177c0aa470b9e94c2714d561db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.488751 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1be8f8557b9a4ae2917c37b8d01cedf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.494865 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 8e17e1ca404b462da43aa1db7dc3b556 (DEFAULT_TABLE table=default.Table_6a8ea07f0b174864b81addf035ed1f34 [id=dfff0b62abf54558b0b04a3b5d8cd896]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.495239 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e17e1ca404b462da43aa1db7dc3b556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.506093 339 tablet_bootstrap.cc:492] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:32.507484 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 1be8f8557b9a4ae2917c37b8d01cedf6 (DEFAULT_TABLE table=default.Table_e8b3bfa63ee7407ead7f4a0f8204d269 [id=02411177c0aa470b9e94c2714d561db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.507929 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1be8f8557b9a4ae2917c37b8d01cedf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.516264 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7 (DEFAULT_TABLE table=default.Table_35a9f061c58a4dd1b8ad6e194d32bbde [id=764330c0798042d3a0ca1e6d0f1ec78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.516736 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.506917 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7 (DEFAULT_TABLE table=default.Table_35a9f061c58a4dd1b8ad6e194d32bbde [id=764330c0798042d3a0ca1e6d0f1ec78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.518630 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.507560 32350 tablet_service.cc:1468] Processing CreateTablet for tablet a8945b6f49474648ac73bcb85285680a (DEFAULT_TABLE table=default.Table_ac62bae678de4b11a971f7bbccd50303 [id=4f4663a5cad14a3593e16a52ec8aeece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.523125 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8945b6f49474648ac73bcb85285680a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.524066 32488 tablet_service.cc:1468] Processing CreateTablet for tablet d6fdea59f71e4a9d9fca8ede4b9439d2 (DEFAULT_TABLE table=default.Table_b9220275c4064c11bb0618d6a459aab6 [id=392754cfbca449d887b33eb6ec5f0ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.524439 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fdea59f71e4a9d9fca8ede4b9439d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.508106 32359 tablet_service.cc:1468] Processing CreateTablet for tablet d6fdea59f71e4a9d9fca8ede4b9439d2 (DEFAULT_TABLE table=default.Table_b9220275c4064c11bb0618d6a459aab6 [id=392754cfbca449d887b33eb6ec5f0ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.526690 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fdea59f71e4a9d9fca8ede4b9439d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.529237 1113 tablet.cc:1722] T 2414d24ab6c54f36bb1e554f7231a5d7 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:32.508566 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet b80a91ae2c084020b5d3444b570b28f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:32.530858 32489 tablet_service.cc:1468] Processing CreateTablet for tablet a8945b6f49474648ac73bcb85285680a (DEFAULT_TABLE table=default.Table_ac62bae678de4b11a971f7bbccd50303 [id=4f4663a5cad14a3593e16a52ec8aeece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.531237 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8945b6f49474648ac73bcb85285680a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.508929 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f9e0d1b85a94ae9ae86e53688a55db5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:32.535135 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 8e17e1ca404b462da43aa1db7dc3b556 (DEFAULT_TABLE table=default.Table_6a8ea07f0b174864b81addf035ed1f34 [id=dfff0b62abf54558b0b04a3b5d8cd896]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.535575 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e17e1ca404b462da43aa1db7dc3b556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.509315 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38ecd4c243b840f4b66d8e01078dca64" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:32.515664 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 3740dd120745493688eeb4c1ac311dcb (DEFAULT_TABLE table=default.Table_7202dd9a46994e949c2368ac8f51b53d [id=4d97ab7c1137414ca359d29f66e8497b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.541121 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3740dd120745493688eeb4c1ac311dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.542954 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet b80a91ae2c084020b5d3444b570b28f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:32.543251 343 raft_consensus.cc:357] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.543901 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0773fc60d4a463596f536f15d58bf57" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:32.544363 339 ts_tablet_manager.cc:1397] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.001s
I20250623 08:04:32.552249 981 ts_tablet_manager.cc:1918] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:32.553435 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 32) took 1598 ms. Trace:
I20250623 08:04:32.548383 979 ts_tablet_manager.cc:1918] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:32.562953 819 raft_consensus.cc:491] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.566241 819 raft_consensus.cc:513] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.566582 343 raft_consensus.cc:383] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.568447 1136 raft_consensus.cc:491] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.568652 1136 raft_consensus.cc:513] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:32.570092 952 rpcz_store.cc:276] 0623 08:04:30.955324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.955402 (+ 78us) service_pool.cc:225] Handling call
0623 08:04:32.252050 (+1296648us) ts_tablet_manager.cc:1145] Deleting tablet ea702abb31fb41498ab61bbb05b1f7e2
0623 08:04:32.252062 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.363386 (+111324us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.471553 (+108167us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.553349 (+ 81796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1296266}
I20250623 08:04:32.570554 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0ee5a4fe03848229524cc0105226d8c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
W20250623 08:04:32.588976 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 154945us
I20250623 08:04:32.589140 32654 raft_consensus.cc:3058] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:32.589180 819 leader_election.cc:290] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:32.589214 343 raft_consensus.cc:738] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.589524 32589 leader_election.cc:304] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.589478 343 consensus_queue.cc:260] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.589764 1136 leader_election.cc:290] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:32.589924 1136 raft_consensus.cc:2802] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:32.589987 32589 leader_election.cc:304] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.590075 1136 raft_consensus.cc:695] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:32.590421 32589 leader_election.cc:304] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.590725 1128 raft_consensus.cc:2802] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:32.590871 1128 raft_consensus.cc:695] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:32.590996 343 ts_tablet_manager.cc:1428] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.137s user 0.003s sys 0.000s
I20250623 08:04:32.591138 1128 consensus_queue.cc:237] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.591321 343 tablet_bootstrap.cc:492] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:32.590481 979 log.cc:1199] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/ea702abb31fb41498ab61bbb05b1f7e2
I20250623 08:04:32.601727 1011 raft_consensus.cc:2747] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:32.603231 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 4ac9f38877b24d619c280a1aaa0b7b08 (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]) successfully deleted
W20250623 08:04:32.603534 32589 leader_election.cc:343] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 04eda92e14144733ab71a577dd3bd2b0
W20250623 08:04:32.603624 32043 catalog_manager.cc:5001] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): alter failed for tablet 4ac9f38877b24d619c280a1aaa0b7b08 (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]),no further retry: Not found: Tablet not found: 4ac9f38877b24d619c280a1aaa0b7b08
I20250623 08:04:32.590397 1136 consensus_queue.cc:237] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.606487 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:32.606709 32655 raft_consensus.cc:3058] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:32.591781 342 tablet_bootstrap.cc:492] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:32.610477 32050 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e026a79cbd6c44efae4e91ab3db8e5fa" } modify_external_catalogs: true
I20250623 08:04:32.602309 1071 tablet.cc:1722] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:32.610677 32050 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e026a79cbd6c44efae4e91ab3db8e5fa" } modify_external_catalogs: true
I20250623 08:04:32.602492 1058 tablet.cc:1722] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:32.611346 342 ts_tablet_manager.cc:1397] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.373s user 0.018s sys 0.030s
I20250623 08:04:32.611923 32056 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6e40e423312e4f9e95174f277b8897ce" } modify_external_catalogs: true
I20250623 08:04:32.612064 952 tablet_replica.cc:331] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:32.612082 32056 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6e40e423312e4f9e95174f277b8897ce" } modify_external_catalogs: true
I20250623 08:04:32.612279 952 raft_consensus.cc:2241] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.612478 952 raft_consensus.cc:2270] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.613005 952 ts_tablet_manager.cc:1905] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.613521 32655 raft_consensus.cc:2466] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 2.
I20250623 08:04:32.613742 1121 tablet.cc:1722] T 31a67b70cbc64bb3bfd5a34865109a01 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:32.615792 342 raft_consensus.cc:357] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.615967 342 raft_consensus.cc:383] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.616027 342 raft_consensus.cc:738] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.616068 343 tablet_bootstrap.cc:654] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.616281 342 consensus_queue.cc:260] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.616437 32654 raft_consensus.cc:2466] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 2.
I20250623 08:04:32.617431 342 ts_tablet_manager.cc:1428] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:32.617751 342 tablet_bootstrap.cc:492] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:32.619400 342 tablet_bootstrap.cc:654] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.624161 979 ts_tablet_manager.cc:1939] T ea702abb31fb41498ab61bbb05b1f7e2 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:32.625236 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 47) took 1750 ms. Trace:
I20250623 08:04:32.625362 979 rpcz_store.cc:276] 0623 08:04:30.874633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.874667 (+ 34us) service_pool.cc:225] Handling call
0623 08:04:32.304657 (+1429990us) ts_tablet_manager.cc:1145] Deleting tablet ea702abb31fb41498ab61bbb05b1f7e2
0623 08:04:32.304816 (+ 159us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.354908 (+ 50092us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.548214 (+193306us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.625141 (+ 76927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429733}
I20250623 08:04:32.625991 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 49) took 1750 ms. Trace:
I20250623 08:04:32.626088 979 rpcz_store.cc:276] 0623 08:04:30.875231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.875273 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:32.625791 (+1750518us) ts_tablet_manager.cc:1145] Deleting tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3
0623 08:04:32.625804 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.625936 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750277}
I20250623 08:04:32.626437 979 tablet_replica.cc:331] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:32.626681 979 raft_consensus.cc:2241] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:32.628033 342 tablet_bootstrap.cc:492] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:32.631131 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet ea702abb31fb41498ab61bbb05b1f7e2 (table default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]) successfully deleted
W20250623 08:04:32.631521 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 already in progress: creating tablet
I20250623 08:04:32.628365 944 consensus_queue.cc:1035] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250623 08:04:32.634771 979 raft_consensus.cc:2270] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.635336 979 ts_tablet_manager.cc:1905] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.634471 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:32.636694 32515 raft_consensus.cc:3058] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:32.639768 32515 raft_consensus.cc:2466] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 2.
I20250623 08:04:32.628187 342 ts_tablet_manager.cc:1397] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250623 08:04:32.640817 342 raft_consensus.cc:357] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.641419 342 raft_consensus.cc:383] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.642161 342 raft_consensus.cc:738] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.642467 342 consensus_queue.cc:260] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.643982 342 ts_tablet_manager.cc:1428] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:32.645555 342 tablet_bootstrap.cc:492] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:32.555876 1123 tablet.cc:1722] T 0861107ab4624a05bd7a4d24b420c502 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:32.557601 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ecf4af9e476495683bd63cac20d4672" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:32.558046 32375 raft_consensus.cc:3058] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:32.647756 342 tablet_bootstrap.cc:654] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:32.628207 32589 leader_election.cc:343] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.648133 32589 leader_election.cc:304] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:32.648607 343 tablet_bootstrap.cc:492] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:32.648778 1128 raft_consensus.cc:2747] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:32.648785 343 ts_tablet_manager.cc:1397] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20250623 08:04:32.649706 343 raft_consensus.cc:357] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.649981 343 raft_consensus.cc:383] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.650092 343 raft_consensus.cc:738] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.650405 1128 raft_consensus.cc:491] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.558374 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:32.650914 1128 raft_consensus.cc:513] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.559163 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d8f38a6de1e45e7bed4abdced684c92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:32.651350 32381 raft_consensus.cc:2391] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 2: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:32.546407 339 raft_consensus.cc:357] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.651706 339 raft_consensus.cc:383] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.651823 339 raft_consensus.cc:738] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.652086 339 consensus_queue.cc:260] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.653033 342 tablet_bootstrap.cc:492] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:32.653183 342 ts_tablet_manager.cc:1397] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20250623 08:04:32.653453 339 ts_tablet_manager.cc:1428] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.109s user 0.001s sys 0.001s
I20250623 08:04:32.650640 343 consensus_queue.cc:260] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.653769 339 tablet_bootstrap.cc:492] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:32.654084 1128 leader_election.cc:290] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:32.654429 343 ts_tablet_manager.cc:1428] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:32.654964 343 tablet_bootstrap.cc:492] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:32.657541 343 tablet_bootstrap.cc:654] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.659561 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712abfca1fa4489f882534c33b9db1be" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:32.660395 32589 leader_election.cc:343] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.662643 339 tablet_bootstrap.cc:654] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.577986 32353 tablet_service.cc:1468] Processing CreateTablet for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (DEFAULT_TABLE table=default.Table_37287a2bae6a4d72ab9a957a92be554b [id=d5366c10bade4a2b93f0a2b2524517fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.595155 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdec6e9573b041e0a57d1e0e0239d97f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:32.604694 1118 tablet.cc:1722] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:32.619096 32314 leader_election.cc:343] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: d0ee5a4fe03848229524cc0105226d8c
I20250623 08:04:32.658649 342 raft_consensus.cc:357] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.665709 342 raft_consensus.cc:383] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.665797 342 raft_consensus.cc:738] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.665843 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7f3d2e67e84bfcbd45a21658f69e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.666044 342 consensus_queue.cc:260] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.667672 342 ts_tablet_manager.cc:1428] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250623 08:04:32.669117 342 tablet_bootstrap.cc:492] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:32.669688 848 consensus_queue.cc:1035] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:32.669759 32453 leader_election.cc:343] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 2ecf4af9e476495683bd63cac20d4672
I20250623 08:04:32.670301 32453 leader_election.cc:304] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:32.670382 848 consensus_queue.cc:1035] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.670720 848 raft_consensus.cc:2762] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20250623 08:04:32.670809 32453 consensus_peers.cc:487] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e47b10f46b8402e94354ebca573c194. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:32.671998 32483 tablet_service.cc:1468] Processing CreateTablet for tablet 3740dd120745493688eeb4c1ac311dcb (DEFAULT_TABLE table=default.Table_7202dd9a46994e949c2368ac8f51b53d [id=4d97ab7c1137414ca359d29f66e8497b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.672501 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3740dd120745493688eeb4c1ac311dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.674006 979 ts_tablet_manager.cc:1918] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:32.680680 342 tablet_bootstrap.cc:654] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.682068 979 log.cc:1199] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/966a159e212040109e2842c15b69fb71
I20250623 08:04:32.682623 979 ts_tablet_manager.cc:1939] T 966a159e212040109e2842c15b69fb71 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:32.683213 32589 leader_election.cc:343] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.683874 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 48) took 1808 ms. Trace:
W20250623 08:04:32.684085 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 46176.7us
I20250623 08:04:32.684289 979 rpcz_store.cc:276] 0623 08:04:30.874943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.875539 (+ 596us) service_pool.cc:225] Handling call
0623 08:04:32.626344 (+1750805us) ts_tablet_manager.cc:1145] Deleting tablet 966a159e212040109e2842c15b69fb71
0623 08:04:32.626356 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.654133 (+ 27777us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.673846 (+ 19713us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.683839 (+ 9993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750591}
I20250623 08:04:32.684631 610 consensus_queue.cc:1035] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.688938 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 50) took 1813 ms. Trace:
I20250623 08:04:32.691108 979 rpcz_store.cc:276] 0623 08:04:30.875534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.875770 (+ 236us) service_pool.cc:225] Handling call
0623 08:04:32.688776 (+1813006us) ts_tablet_manager.cc:1145] Deleting tablet 8dc696236a15420e9666cca708068fb2
0623 08:04:32.688786 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.688909 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1812787}
I20250623 08:04:32.692886 979 tablet_replica.cc:331] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:32.693173 979 raft_consensus.cc:2241] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.693434 981 log.cc:1199] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/0311ce67989f49e8866234a88e5b4d09
I20250623 08:04:32.694051 981 ts_tablet_manager.cc:1939] T 0311ce67989f49e8866234a88e5b4d09 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:32.694900 979 raft_consensus.cc:2270] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.695638 979 ts_tablet_manager.cc:1905] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:32.696038 32043 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.60:39691, user_credentials={real_user=slave}} blocked reactor thread for 48108.3us
I20250623 08:04:32.696343 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 966a159e212040109e2842c15b69fb71 (table default.Table_84c686e72347490ebc29749f8c9352f9 [id=27884529aa7042ffb7eb7812892c6305]) successfully deleted
W20250623 08:04:32.696671 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 8dc696236a15420e9666cca708068fb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dc696236a15420e9666cca708068fb2 already in progress: creating tablet
I20250623 08:04:32.699810 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 23) took 1955 ms. Trace:
W20250623 08:04:32.694212 32314 leader_election.cc:343] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: ce1f3ef3690c4837b95b105733884f1f
I20250623 08:04:32.701161 32314 leader_election.cc:304] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:32.707698 981 rpcz_store.cc:276] 0623 08:04:30.744463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.745396 (+ 933us) service_pool.cc:225] Handling call
0623 08:04:32.309553 (+1564157us) ts_tablet_manager.cc:1145] Deleting tablet 0311ce67989f49e8866234a88e5b4d09
0623 08:04:32.309566 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.455286 (+145720us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.552111 (+ 96825us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.699755 (+147644us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563845}
I20250623 08:04:32.708304 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 24) took 1962 ms. Trace:
I20250623 08:04:32.708472 981 rpcz_store.cc:276] 0623 08:04:30.745903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.746835 (+ 932us) service_pool.cc:225] Handling call
0623 08:04:32.708129 (+1961294us) ts_tablet_manager.cc:1145] Deleting tablet 965f1f0a107b4653b6aac01e0d8e259a
0623 08:04:32.708139 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.708250 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1961050}
I20250623 08:04:32.708935 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 25) took 1961 ms. Trace:
I20250623 08:04:32.709065 981 rpcz_store.cc:276] 0623 08:04:30.747889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.748004 (+ 115us) service_pool.cc:225] Handling call
0623 08:04:32.708793 (+1960789us) ts_tablet_manager.cc:1145] Deleting tablet d0b2af698b6e4ed0ae66a9bb32be7b31
0623 08:04:32.708801 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.708887 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1960595}
I20250623 08:04:32.709388 1001 raft_consensus.cc:2747] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:32.709875 1001 consensus_queue.cc:1035] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.710362 1001 consensus_queue.cc:1035] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.711571 1062 raft_consensus.cc:491] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:32.711774 1062 raft_consensus.cc:513] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.712652 1062 leader_election.cc:290] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:32.718775 32050 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7404f0e4240a4122af62139768eb9367
I20250623 08:04:32.719646 339 tablet_bootstrap.cc:492] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:32.719914 339 ts_tablet_manager.cc:1397] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.007s
I20250623 08:04:32.720948 339 raft_consensus.cc:357] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.721237 339 raft_consensus.cc:383] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.721371 339 raft_consensus.cc:738] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.721666 339 consensus_queue.cc:260] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.722201 339 ts_tablet_manager.cc:1428] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:32.722646 339 tablet_bootstrap.cc:492] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:32.723618 343 tablet_bootstrap.cc:492] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:32.723879 343 ts_tablet_manager.cc:1397] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.069s user 0.002s sys 0.007s
I20250623 08:04:32.724928 343 raft_consensus.cc:357] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.725229 343 raft_consensus.cc:383] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.725378 343 raft_consensus.cc:738] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.725771 343 consensus_queue.cc:260] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.726392 343 ts_tablet_manager.cc:1428] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250623 08:04:32.730953 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 59914.3us
I20250623 08:04:32.741617 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet ea702abb31fb41498ab61bbb05b1f7e2 (table default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]) successfully deleted
I20250623 08:04:32.745397 343 tablet_bootstrap.cc:492] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:32.726665 1001 consensus_queue.cc:1035] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:32.747304 339 tablet_bootstrap.cc:654] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.747365 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_849f3be400444579943f3f649824f4ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.748083 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_849f3be400444579943f3f649824f4ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.749727 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_5338d405e4ed40589c691093043355d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.750250 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5338d405e4ed40589c691093043355d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.752130 343 tablet_bootstrap.cc:654] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:32.738998 32314 leader_election.cc:343] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: d0ee5a4fe03848229524cc0105226d8c
I20250623 08:04:32.754945 32314 leader_election.cc:304] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:32.759740 979 ts_tablet_manager.cc:1918] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:32.764281 952 ts_tablet_manager.cc:1918] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:32.764689 979 log.cc:1199] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/162308da898d4dde8ab5b1ab01d0f393
I20250623 08:04:32.765169 979 ts_tablet_manager.cc:1939] T 162308da898d4dde8ab5b1ab01d0f393 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:32.766007 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 51) took 1890 ms. Trace:
I20250623 08:04:32.772025 952 log.cc:1199] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/4cb66c7a273a412db3ae77af9e30d036
I20250623 08:04:32.772285 343 tablet_bootstrap.cc:492] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:32.772459 979 rpcz_store.cc:276] 0623 08:04:30.875819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.876015 (+ 196us) service_pool.cc:225] Handling call
0623 08:04:32.692823 (+1816808us) ts_tablet_manager.cc:1145] Deleting tablet 162308da898d4dde8ab5b1ab01d0f393
0623 08:04:32.692832 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.716747 (+ 23915us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.759610 (+ 42863us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.765953 (+ 6343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1816569}
I20250623 08:04:32.772635 343 ts_tablet_manager.cc:1397] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.001s
I20250623 08:04:32.772744 952 ts_tablet_manager.cc:1939] T 4cb66c7a273a412db3ae77af9e30d036 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:32.773155 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 62) took 1763 ms. Trace:
I20250623 08:04:32.773314 979 rpcz_store.cc:276] 0623 08:04:31.009379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.009431 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:32.773000 (+1763569us) ts_tablet_manager.cc:1145] Deleting tablet 4ac9f38877b24d619c280a1aaa0b7b08
0623 08:04:32.773012 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.773098 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1763337}
I20250623 08:04:32.773666 343 raft_consensus.cc:357] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.773950 343 raft_consensus.cc:383] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.774075 343 raft_consensus.cc:738] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.774071 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 35) took 1817 ms. Trace:
I20250623 08:04:32.774192 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 162308da898d4dde8ab5b1ab01d0f393 (table default.Table_c30b48097bd643b18b2a744839780b9e [id=7e59ad2235c1418d97f2b73ff2a48300]) successfully deleted
I20250623 08:04:32.774187 952 rpcz_store.cc:276] 0623 08:04:30.956084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.956128 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:32.611979 (+1655851us) ts_tablet_manager.cc:1145] Deleting tablet 4cb66c7a273a412db3ae77af9e30d036
0623 08:04:32.611989 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.677727 (+ 65738us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.764118 (+ 86391us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.774027 (+ 9909us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655554}
I20250623 08:04:32.774433 343 consensus_queue.cc:260] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:32.774519 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 4ac9f38877b24d619c280a1aaa0b7b08 because the tablet was not found. No further retry: Not found: Tablet not found: 4ac9f38877b24d619c280a1aaa0b7b08
I20250623 08:04:32.775074 343 ts_tablet_manager.cc:1428] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:32.775761 343 tablet_bootstrap.cc:492] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:32.776177 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 63) took 1766 ms. Trace:
I20250623 08:04:32.776342 979 rpcz_store.cc:276] 0623 08:04:31.009821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.009865 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:32.776015 (+1766150us) ts_tablet_manager.cc:1145] Deleting tablet 205da880685b49b3b8669f73dc2c563b
0623 08:04:32.776024 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.776116 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1765936}
I20250623 08:04:32.778338 610 raft_consensus.cc:491] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.778234 1062 consensus_queue.cc:1035] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
W20250623 08:04:32.778859 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 205da880685b49b3b8669f73dc2c563b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 205da880685b49b3b8669f73dc2c563b already in progress: creating tablet
W20250623 08:04:32.778903 32314 consensus_peers.cc:487] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32c82141edea4068b54755a85c6d7fb8. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:32.778877 1001 raft_consensus.cc:2802] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:32.779109 1001 raft_consensus.cc:695] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:32.779194 32314 leader_election.cc:304] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:32.779265 979 tablet_replica.cc:331] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:32.779458 979 raft_consensus.cc:2241] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.779624 979 raft_consensus.cc:2270] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.780153 979 ts_tablet_manager.cc:1905] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.780207 952 tablet_replica.cc:331] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:32.780802 952 raft_consensus.cc:2241] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:32.782145 892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 08:04:32.782613 892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 08:04:32.782784 892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250623 08:04:32.783397 952 raft_consensus.cc:2270] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.784281 952 ts_tablet_manager.cc:1905] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.787400 343 tablet_bootstrap.cc:654] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.787954 1065 raft_consensus.cc:2747] T 31a67b70cbc64bb3bfd5a34865109a01 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:32.788479 657 raft_consensus.cc:491] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.788012 610 raft_consensus.cc:513] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.788686 657 raft_consensus.cc:513] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.789400 657 leader_election.cc:290] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:32.789322 1001 consensus_queue.cc:237] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:32.790535 32314 consensus_peers.cc:487] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 228375592aee48a1aa21e1d16962b663. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:32.790831 32314 consensus_peers.cc:487] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7da84969a6464b5caeaf47af0cbaaa4d. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:32.791194 32314 consensus_peers.cc:487] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67c44697ecab4dd9b43f544a602897ca. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:32.792690 1118 tablet.cc:1722] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:32.793018 610 leader_election.cc:290] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:32.807886 339 tablet_bootstrap.cc:492] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:32.808091 339 ts_tablet_manager.cc:1397] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.005s
I20250623 08:04:32.808982 339 raft_consensus.cc:357] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.809198 339 raft_consensus.cc:383] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.809293 339 raft_consensus.cc:738] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.809407 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_49e1ee7e0d024625988be24a823a443c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:32.809576 339 consensus_queue.cc:260] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.810253 339 ts_tablet_manager.cc:1428] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250623 08:04:32.810581 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49e1ee7e0d024625988be24a823a443c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.813297 979 ts_tablet_manager.cc:1918] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:32.813512 979 log.cc:1199] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/8d8f38a6de1e45e7bed4abdced684c92
I20250623 08:04:32.813663 32314 leader_election.cc:304] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:32.813983 979 ts_tablet_manager.cc:1939] T 8d8f38a6de1e45e7bed4abdced684c92 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:32.814006 746 raft_consensus.cc:2802] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:32.814218 746 raft_consensus.cc:695] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:32.814560 746 consensus_queue.cc:237] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.821467 657 raft_consensus.cc:491] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.821698 657 raft_consensus.cc:513] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.822597 657 leader_election.cc:290] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:32.822719 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 64) took 1812 ms. Trace:
I20250623 08:04:32.822911 979 rpcz_store.cc:276] 0623 08:04:31.010301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.010357 (+ 56us) service_pool.cc:225] Handling call
0623 08:04:32.779174 (+1768817us) ts_tablet_manager.cc:1145] Deleting tablet 8d8f38a6de1e45e7bed4abdced684c92
0623 08:04:32.779185 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.787830 (+ 8645us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.813108 (+ 25278us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.822678 (+ 9570us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1768641}
I20250623 08:04:32.823340 979 tablet_replica.cc:331] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:32.823558 343 tablet_bootstrap.cc:492] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:32.823504 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 8d8f38a6de1e45e7bed4abdced684c92 (table default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e [id=7a569a9fe3bd4c608ddca48a734df1da]) successfully deleted
I20250623 08:04:32.823604 979 raft_consensus.cc:2241] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.823894 979 raft_consensus.cc:2270] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.709424 981 tablet_replica.cc:331] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:32.824476 979 ts_tablet_manager.cc:1905] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.824811 981 raft_consensus.cc:2241] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.825063 981 pending_rounds.cc:70] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:32.825198 981 pending_rounds.cc:77] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727400669351936 op_type: NO_OP noop_request { }
I20250623 08:04:32.825479 981 raft_consensus.cc:2887] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:32.825698 981 raft_consensus.cc:2270] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.826900 339 tablet_bootstrap.cc:492] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:32.831521 979 ts_tablet_manager.cc:1918] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:32.831686 979 log.cc:1199] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/35ab9f77a81a4490abb0d2c0c69cd90f
I20250623 08:04:32.832072 979 ts_tablet_manager.cc:1939] T 35ab9f77a81a4490abb0d2c0c69cd90f P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:32.832441 343 ts_tablet_manager.cc:1397] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.000s
I20250623 08:04:32.835515 343 raft_consensus.cc:357] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:32.835734 32453 leader_election.cc:343] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.835733 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 0311ce67989f49e8866234a88e5b4d09 (table default.Table_e962b9945a5041039f7e33b98b653175 [id=9d9e232100264e378516c06c61310750]) successfully deleted
W20250623 08:04:32.836089 32453 leader_election.cc:343] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:32.836141 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 965f1f0a107b4653b6aac01e0d8e259a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 965f1f0a107b4653b6aac01e0d8e259a already in progress: creating tablet
I20250623 08:04:32.836263 343 raft_consensus.cc:383] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 08:04:32.836520 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0b2af698b6e4ed0ae66a9bb32be7b31 already in progress: creating tablet
I20250623 08:04:32.836690 343 raft_consensus.cc:738] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.836953 343 consensus_queue.cc:260] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.839356 343 ts_tablet_manager.cc:1428] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:32.841423 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_da6195995899479b879a3d41e0c07ed4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.842284 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da6195995899479b879a3d41e0c07ed4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.843456 1118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:35542 (request call id 41) took 1608 ms. Trace:
I20250623 08:04:32.844658 1136 raft_consensus.cc:491] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.844839 1136 raft_consensus.cc:513] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.845640 1136 leader_election.cc:290] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:32.846025 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6ed4cb252b1452087f8ee53ec78d985" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:32.846624 32589 leader_election.cc:343] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:32.855453 892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 08:04:32.855638 892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 08:04:32.855770 892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 08:04:32.855863 892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 08:04:32.857933 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "432d1642c0e44119b5bc05858463a136" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:32.858536 343 tablet_bootstrap.cc:492] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:32.858800 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 65) took 1848 ms. Trace:
I20250623 08:04:32.858917 979 rpcz_store.cc:276] 0623 08:04:31.010705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.010762 (+ 57us) service_pool.cc:225] Handling call
0623 08:04:32.823257 (+1812495us) ts_tablet_manager.cc:1145] Deleting tablet 35ab9f77a81a4490abb0d2c0c69cd90f
0623 08:04:32.823266 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.827779 (+ 4513us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.831400 (+ 3621us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.858713 (+ 27313us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21,"tablet-delete.queue_time_us":1812297}
I20250623 08:04:32.859464 979 tablet_replica.cc:331] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:32.859747 979 raft_consensus.cc:2241] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:32.860173 979 raft_consensus.cc:2270] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.860416 343 tablet_bootstrap.cc:654] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.860890 979 ts_tablet_manager.cc:1905] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.863849 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 35ab9f77a81a4490abb0d2c0c69cd90f (table default.Table_92e23e1081544f8c9b99f14654981494 [id=bbf8e66e745141348c56d1cea5716898]) successfully deleted
I20250623 08:04:32.866101 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "432d1642c0e44119b5bc05858463a136" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:32.875636 343 tablet_bootstrap.cc:492] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:32.875891 343 ts_tablet_manager.cc:1397] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250623 08:04:32.876664 343 raft_consensus.cc:357] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.876968 343 raft_consensus.cc:383] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.877084 343 raft_consensus.cc:738] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.877553 343 consensus_queue.cc:260] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.878271 343 ts_tablet_manager.cc:1428] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:32.878661 343 tablet_bootstrap.cc:492] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:32.880429 343 tablet_bootstrap.cc:654] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.880924 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_f40b881c21294c10b35962b75a30524c" } new_table_name: "default.Table_8649c62e786b4a0cab61befe1c6c7ec7" modify_external_catalogs: true
I20250623 08:04:32.882297 892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.69.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.61:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.69.61
--webserver_port=44043
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 07:50:47 UTC on 24a791456cd2
build id 6728
ASAN enabled
I20250623 08:04:32.884423 979 ts_tablet_manager.cc:1918] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:32.884713 892 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250623 08:04:32.889917 892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250623 08:04:32.894232 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_fb2522ad8b664fdc9ec779b46b807ca1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.895236 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb2522ad8b664fdc9ec779b46b807ca1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:32.896406 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adc3cf4355c24b7395fcb22af8520346" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:32.898218 342 tablet_bootstrap.cc:492] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:32.898386 342 ts_tablet_manager.cc:1397] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.229s user 0.006s sys 0.017s
I20250623 08:04:32.899264 342 raft_consensus.cc:357] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.899468 342 raft_consensus.cc:383] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.899564 342 raft_consensus.cc:738] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.899797 342 consensus_queue.cc:260] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.900372 342 ts_tablet_manager.cc:1428] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:32.900720 342 tablet_bootstrap.cc:492] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:32.904547 339 tablet_bootstrap.cc:654] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.904603 746 raft_consensus.cc:491] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:32.904839 746 raft_consensus.cc:513] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:32.905714 746 leader_election.cc:290] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:32.905918 1118 rpcz_store.cc:276] 0623 08:04:31.235473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.235523 (+ 50us) service_pool.cc:225] Handling call
0623 08:04:32.843418 (+1607895us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:31.259745 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:31.309721 (+ 49976us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:31.309734 (+ 13us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:31.309744 (+ 10us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:31.309796 (+ 52us) alter_schema_op.cc:128] START. Timestamp: 1750665871309730
0623 08:04:31.309801 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:31.522704 (+212903us) log.cc:844] Serialized 173 byte log entry
0623 08:04:31.534907 (+ 12203us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x6160001a1a54. stack: 00007f7f6e5c4d56 00007f7f6e5c4b10 00007f7f6d5bb29d 0000000000970a84 0000000000970a4d 000000000097041a 00007f7f79ff85e2 00007f7f7b39f255 00007f7f7b39dd79 00007f7f7b3a61a0 00007f7f7b3a5fad 0000000000a751bc 00007f7f6e60a3ff 00007f7f6e60d6cb 00007f7f6e60d548 0000000000a751bc
0623 08:04:32.792470 (+1257563us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.792606 (+ 136us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:32.793102 (+ 496us) log.cc:844] Serialized 5 byte log entry
0623 08:04:32.842960 (+ 49858us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.843339 (+ 379us) log.cc:844] Serialized 14 byte log entry
0623 08:04:32.843376 (+ 37us) alter_schema_op.cc:73] Released schema lock
0623 08:04:32.843380 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"prepare.queue_time_us":320,"prepare.run_cpu_time_us":2933,"prepare.run_wall_time_us":275195,"replication_time_us":1482628,"spinlock_wait_cycles":375176576,"thread_start_us":47368,"threads_started":1,"wal-append.queue_time_us":29566}]]}
I20250623 08:04:32.915840 981 ts_tablet_manager.cc:1905] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.897328 343 tablet_bootstrap.cc:492] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:32.918987 979 log.cc:1199] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/9bc95f12ab5d4b9b9f0729cb88aabbaa
I20250623 08:04:32.919032 343 ts_tablet_manager.cc:1397] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.007s
I20250623 08:04:32.898845 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adc3cf4355c24b7395fcb22af8520346" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:32.919904 343 raft_consensus.cc:357] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.920089 343 raft_consensus.cc:383] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.920163 343 raft_consensus.cc:738] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:32.920377 343 consensus_queue.cc:260] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.920881 343 ts_tablet_manager.cc:1428] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:32.921201 343 tablet_bootstrap.cc:492] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:32.923317 343 tablet_bootstrap.cc:654] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.928478 1121 tablet.cc:1722] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:32.929183 979 ts_tablet_manager.cc:1939] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:32.934849 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 66) took 1923 ms. Trace:
I20250623 08:04:32.937121 32381 raft_consensus.cc:3058] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:32.939514 979 rpcz_store.cc:276] 0623 08:04:31.011105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.011149 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:32.859352 (+1848203us) ts_tablet_manager.cc:1145] Deleting tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa
0623 08:04:32.859365 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.869064 (+ 9699us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.884250 (+ 15186us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:32.934810 (+ 50560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1848005}
I20250623 08:04:32.940414 979 tablet_replica.cc:331] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:32.940663 979 raft_consensus.cc:2241] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:32.940846 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]) successfully deleted
I20250623 08:04:32.941989 979 raft_consensus.cc:2270] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:32.950348 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_dd7b43110df2433597689f77b73907a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:32.951303 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd7b43110df2433597689f77b73907a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:32.953632 1193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:32.968067 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:32.955232 343 tablet_bootstrap.cc:492] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:32.963013 32632 tablet_service.cc:1468] Processing CreateTablet for tablet 90037c523a1f4118af268e088bb97062 (DEFAULT_TABLE table=default.Table_5cf87169ed0d4987b396195d518aa055 [id=5edc67ced51e4cde9dbfcf448d24b87d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.968611 343 ts_tablet_manager.cc:1397] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.008s
I20250623 08:04:32.968804 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90037c523a1f4118af268e088bb97062. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.969990 343 raft_consensus.cc:357] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.970242 343 raft_consensus.cc:383] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:32.970322 343 raft_consensus.cc:738] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:32.970494 1197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:32.964037 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 803d3607d75443568b7ddc9f0c0bdc90 (DEFAULT_TABLE table=default.Table_98f40c99d9ff4819b958cd84cc5dc3a9 [id=13cf327a200c490992163a70f5a275cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.970630 343 consensus_queue.cc:260] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:32.971114 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803d3607d75443568b7ddc9f0c0bdc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.971181 343 ts_tablet_manager.cc:1428] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:32.971478 343 tablet_bootstrap.cc:492] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:32.971719 892 server_base.cc:1048] running on GCE node
I20250623 08:04:32.972934 32375 raft_consensus.cc:3058] T 0545daa6fba846adbc573a63d8dbcd66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:32.973564 343 tablet_bootstrap.cc:654] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:32.978842 1136 consensus_queue.cc:1035] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:32.979827 1191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:32.981784 979 ts_tablet_manager.cc:1905] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:32.986299 892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 08:04:32.985745 32052 master_service.cc:948] ReplaceTablet: received request to replace tablet 4c9d1c8a67e149478a0d69842a4eb5b2 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:32.990303 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 (DEFAULT_TABLE table=default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.990944 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfa787f828b411ab68d81bb896f7fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.991397 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f02f36d6c1174a23a1f1043a431d0b6b (table default.Table_75451c4863a34537a328617c53d46cae [id=be88c6ecf20d4ab1b3757c27ce998f87]). Expected version 2 got 0
I20250623 08:04:32.991776 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:32.992090 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:32.992419 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:32.992688 32058 catalog_manager.cc:5582] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:32.993789 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet ba3d2fa01d574dd992dc904ac024ba3a (table default.Table_d496fd69b6f140b88b5abc86c8839ee2 [id=6c57db68f5ed45a3b64d1e3a02ce8a2a]). Expected version 1 got 0
I20250623 08:04:32.995147 32632 tablet_service.cc:1468] Processing CreateTablet for tablet d19d468ce13d4108a9badc36fc503306 (DEFAULT_TABLE table=default.Table_5c43b8626a6640e4a61e495ab2ea5697 [id=46664e505d9648ca889484123253a613]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:32.995692 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d19d468ce13d4108a9badc36fc503306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.995805 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_087d39ad116a4ac2a1cf1050403ef202 [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 1 got 0
W20250623 08:04:32.924079 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 86872.8us
I20250623 08:04:32.997663 892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20250623 08:04:32.998206 32453 consensus_peers.cc:487] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:32.999361 32453 leader_election.cc:343] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:32.999503 32453 leader_election.cc:304] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
W20250623 08:04:32.999707 32453 raft_consensus.cc:2684] T 2ecf4af9e476495683bd63cac20d4672 P 59f7c614254f42009306b13337f06576: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250623 08:04:33.000407 32453 leader_election.cc:343] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.000522 32453 leader_election.cc:304] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.000737 657 raft_consensus.cc:2747] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.003221 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712abfca1fa4489f882534c33b9db1be" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.004985 1071 tablet.cc:1722] T 31a67b70cbc64bb3bfd5a34865109a01 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:33.005914 32589 leader_election.cc:343] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.006078 32589 leader_election.cc:304] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.006590 944 raft_consensus.cc:2747] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.007203 1054 raft_consensus.cc:491] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.007357 1054 raft_consensus.cc:513] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.007934 1054 leader_election.cc:290] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.008356 892 hybrid_clock.cc:648] HybridClock initialized: now 1750665873008252 us; error 79 us; skew 500 ppm
I20250623 08:04:33.008896 892 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 08:04:33.008931 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6ed4cb252b1452087f8ee53ec78d985" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.011683 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 803d3607d75443568b7ddc9f0c0bdc90 (DEFAULT_TABLE table=default.Table_98f40c99d9ff4819b958cd84cc5dc3a9 [id=13cf327a200c490992163a70f5a275cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.012107 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803d3607d75443568b7ddc9f0c0bdc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.012363 32488 tablet_service.cc:1468] Processing CreateTablet for tablet 5443f12974bf41929d9cd937f146cd56 (DEFAULT_TABLE table=default.Table_eda413af2cf84c009c8fc4b375b57e2a [id=898ec01859bf4ba29e38b4eaf3d5392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.012714 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5443f12974bf41929d9cd937f146cd56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.013613 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 (DEFAULT_TABLE table=default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.013968 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfa787f828b411ab68d81bb896f7fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.022528 32493 tablet_service.cc:1468] Processing CreateTablet for tablet d19d468ce13d4108a9badc36fc503306 (DEFAULT_TABLE table=default.Table_5c43b8626a6640e4a61e495ab2ea5697 [id=46664e505d9648ca889484123253a613]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.023005 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d19d468ce13d4108a9badc36fc503306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.026530 32483 tablet_service.cc:1468] Processing CreateTablet for tablet 90037c523a1f4118af268e088bb97062 (DEFAULT_TABLE table=default.Table_5cf87169ed0d4987b396195d518aa055 [id=5edc67ced51e4cde9dbfcf448d24b87d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.027005 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90037c523a1f4118af268e088bb97062. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:32.902732 342 tablet_bootstrap.cc:654] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.033267 339 tablet_bootstrap.cc:492] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:33.033744 339 ts_tablet_manager.cc:1397] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.207s user 0.006s sys 0.010s
I20250623 08:04:33.035492 339 raft_consensus.cc:357] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.035802 339 raft_consensus.cc:383] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.035943 339 raft_consensus.cc:738] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.036330 339 consensus_queue.cc:260] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.037967 339 ts_tablet_manager.cc:1428] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:33.039412 339 tablet_bootstrap.cc:492] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:33.050211 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdec6e9573b041e0a57d1e0e0239d97f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.050604 32517 raft_consensus.cc:2466] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:33.003809 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 4cb66c7a273a412db3ae77af9e30d036 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]) successfully deleted
I20250623 08:04:33.005717 32052 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 4c9d1c8a67e149478a0d69842a4eb5b2
W20250623 08:04:33.052877 32043 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 49039.9us
I20250623 08:04:33.053169 32052 catalog_manager.cc:6652] ReplaceTablet: tablet 4c9d1c8a67e149478a0d69842a4eb5b2 deleted and replaced by tablet 92005abc61dd40ef9da8124cadc29b6a
I20250623 08:04:33.053423 32052 master_service.cc:948] ReplaceTablet: received request to replace tablet 966a159e212040109e2842c15b69fb71 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:33.053843 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 5443f12974bf41929d9cd937f146cd56 (DEFAULT_TABLE table=default.Table_eda413af2cf84c009c8fc4b375b57e2a [id=898ec01859bf4ba29e38b4eaf3d5392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.054301 32624 tablet_service.cc:1468] Processing CreateTablet for tablet 74da35cd34c14284af004001b25aa531 (DEFAULT_TABLE table=default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.054703 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 16331ba6a34942a5a1b4ff4ac7bac55e (DEFAULT_TABLE table=default.Table_4533668b8daa4ff3ad00ccc95e97e032 [id=08d03b1b488c4b52abebcef8b3d0f853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.055085 32634 tablet_service.cc:1468] Processing CreateTablet for tablet affa3da7540648d28c5f00a3cf348eb1 (DEFAULT_TABLE table=default.Table_9c6bc2fbfc274d2e8414205f629cb439 [id=42f65c762e2b46469d3ff189b698b447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.055449 32631 tablet_service.cc:1468] Processing CreateTablet for tablet a0e9b2f810014e81a29ca9b41cd3f8ba (DEFAULT_TABLE table=default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.055490 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 0861107ab4624a05bd7a4d24b420c502 (table default.Table_977fe8fc665f4267b394065e22f67eea [id=be9754ecb40a413d9940224adbd7e2eb]). Expected version 2 got 0
I20250623 08:04:33.055714 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 92005abc61dd40ef9da8124cadc29b6a) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:33.055830 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:33.056121 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5443f12974bf41929d9cd937f146cd56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.056612 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:33.057245 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:33.057611 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:33.058101 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 3 got 0
I20250623 08:04:33.059149 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affa3da7540648d28c5f00a3cf348eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.059607 981 ts_tablet_manager.cc:1918] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.059791 981 log.cc:1199] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/f3c850021e914530bb106ab7166be7c9
I20250623 08:04:33.059988 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "358cb6ac24414bdfbc32155e732f9ac8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:33.060194 981 ts_tablet_manager.cc:1939] T f3c850021e914530bb106ab7166be7c9 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.060225 32655 raft_consensus.cc:2466] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:33.061415 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 26) took 2097 ms. Trace:
I20250623 08:04:33.061506 981 rpcz_store.cc:276] 0623 08:04:30.963984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.964048 (+ 64us) service_pool.cc:225] Handling call
0623 08:04:32.709343 (+1745295us) ts_tablet_manager.cc:1145] Deleting tablet f3c850021e914530bb106ab7166be7c9
0623 08:04:32.709351 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.944013 (+234662us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.059448 (+115435us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.061359 (+ 1911us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745090}
I20250623 08:04:33.061902 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 27) took 2097 ms. Trace:
I20250623 08:04:33.061972 981 rpcz_store.cc:276] 0623 08:04:30.964389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.964436 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:33.061756 (+2097320us) ts_tablet_manager.cc:1145] Deleting tablet f4a0b9402a264800a1ea39707c10da0c
0623 08:04:33.061765 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.061856 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2097078}
I20250623 08:04:33.062254 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 28) took 2097 ms. Trace:
I20250623 08:04:33.062317 981 rpcz_store.cc:276] 0623 08:04:30.964836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.964876 (+ 40us) service_pool.cc:225] Handling call
0623 08:04:33.062151 (+2097275us) ts_tablet_manager.cc:1145] Deleting tablet d6ed4cb252b1452087f8ee53ec78d985
0623 08:04:33.062158 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.062215 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2097124}
I20250623 08:04:33.062654 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 42) took 1825 ms. Trace:
I20250623 08:04:33.062726 981 rpcz_store.cc:276] 0623 08:04:31.237019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.237069 (+ 50us) service_pool.cc:225] Handling call
0623 08:04:33.062508 (+1825439us) ts_tablet_manager.cc:1145] Deleting tablet f3d32798168345edb9898751d083057c
0623 08:04:33.062516 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.062605 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800645}
I20250623 08:04:33.063067 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e9b2f810014e81a29ca9b41cd3f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.067039 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74da35cd34c14284af004001b25aa531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.071619 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16331ba6a34942a5a1b4ff4ac7bac55e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:33.077031 32453 leader_election.cc:343] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 965f1f0a107b4653b6aac01e0d8e259a
I20250623 08:04:33.077404 32453 leader_election.cc:304] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:33.078138 1159 raft_consensus.cc:2802] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:33.078294 1159 raft_consensus.cc:491] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:33.078466 1159 raft_consensus.cc:3058] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.080265 1128 consensus_queue.cc:1035] T 0545daa6fba846adbc573a63d8dbcd66 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:33.082438 1159 raft_consensus.cc:513] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.086019 339 tablet_bootstrap.cc:654] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.086238 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74842fde5152419ca35f3950e465247a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:33.088979 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3d32798168345edb9898751d083057c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:33.089058 32589 leader_election.cc:343] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.089269 32589 leader_election.cc:304] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.089337 32654 raft_consensus.cc:2466] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:33.089709 1136 raft_consensus.cc:2747] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.093902 343 tablet_bootstrap.cc:492] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:33.094122 343 ts_tablet_manager.cc:1397] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.123s user 0.002s sys 0.007s
I20250623 08:04:33.095086 343 raft_consensus.cc:357] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.097074 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (table default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b]). Expected version 1 got 0
I20250623 08:04:33.103349 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
W20250623 08:04:33.103755 32043 catalog_manager.cc:5001] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): alter failed for tablet 0311ce67989f49e8866234a88e5b4d09 (table default.Table_e962b9945a5041039f7e33b98b653175 [id=9d9e232100264e378516c06c61310750]),no further retry: Not found: Tablet not found: 0311ce67989f49e8866234a88e5b4d09
W20250623 08:04:33.104251 32314 consensus_peers.cc:487] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 726d80e05a344f27b5a289723cb5aed7. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:33.111419 1159 leader_election.cc:290] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.090639 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74842fde5152419ca35f3950e465247a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.112833 342 tablet_bootstrap.cc:492] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.113020 342 ts_tablet_manager.cc:1397] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.212s user 0.002s sys 0.005s
I20250623 08:04:33.113883 342 raft_consensus.cc:357] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.114115 342 raft_consensus.cc:383] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.114202 342 raft_consensus.cc:738] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.114449 342 consensus_queue.cc:260] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.115002 342 ts_tablet_manager.cc:1428] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:33.115605 342 tablet_bootstrap.cc:492] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.096073 32618 tablet_service.cc:1468] Processing CreateTablet for tablet bb06a340f82047ac8ec3516edb08f4b6 (DEFAULT_TABLE table=default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.116137 343 raft_consensus.cc:383] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.116263 343 raft_consensus.cc:738] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.116478 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb06a340f82047ac8ec3516edb08f4b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.116545 343 consensus_queue.cc:260] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.117252 343 ts_tablet_manager.cc:1428] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20250623 08:04:33.117614 343 tablet_bootstrap.cc:492] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:33.119626 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:33.120051 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:33.120298 32058 catalog_manager.cc:5582] T 432d1642c0e44119b5bc05858463a136 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:33.120640 339 tablet_bootstrap.cc:492] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:33.120872 32058 catalog_manager.cc:5582] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:33.121114 32632 tablet_service.cc:1468] Processing CreateTablet for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 (DEFAULT_TABLE table=default.Table_5d6ec4f6805b4e0883b8d0419625d033 [id=545b478879aa4f4cb35393e3065d3ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.121376 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:33.121660 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.121703 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_3af33890e8ff40c881af74580254633b [id=8b20894a91c54a3eb08d5e1b4a42d269]). Expected version 3 got 0
I20250623 08:04:33.122130 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:33.122444 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 18ea63263b9b466e922169eecafab747 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]). Expected version 1 got 0
I20250623 08:04:33.122777 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]). Expected version 2 got 0
I20250623 08:04:33.123267 32058 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 31a67b70cbc64bb3bfd5a34865109a01 (table default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e [id=7a569a9fe3bd4c608ddca48a734df1da]). Expected version 2 got 0
I20250623 08:04:33.096478 1136 raft_consensus.cc:491] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 08:04:33.098940 32589 leader_election.cc:343] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 04eda92e14144733ab71a577dd3bd2b0
I20250623 08:04:33.114614 32631 tablet_service.cc:1468] Processing CreateTablet for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (DEFAULT_TABLE table=default.Table_a7cbd31e5c664fba885a13dcbd9105ab [id=86224d949cab4304b8bcd6b429415163]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.125136 32589 leader_election.cc:304] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.125227 1136 raft_consensus.cc:513] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:33.125353 32589 raft_consensus.cc:2684] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:33.125480 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f217a8eb12014fcf89b9df4a2cddf5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.125679 32589 leader_election.cc:304] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.125830 1136 leader_election.cc:290] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:33.125986 1136 raft_consensus.cc:2802] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:33.126071 1136 raft_consensus.cc:491] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:33.126411 1136 raft_consensus.cc:3058] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.127887 32488 tablet_service.cc:1468] Processing CreateTablet for tablet a0e9b2f810014e81a29ca9b41cd3f8ba (DEFAULT_TABLE table=default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.128131 32490 tablet_service.cc:1468] Processing CreateTablet for tablet affa3da7540648d28c5f00a3cf348eb1 (DEFAULT_TABLE table=default.Table_9c6bc2fbfc274d2e8414205f629cb439 [id=42f65c762e2b46469d3ff189b698b447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.128319 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e9b2f810014e81a29ca9b41cd3f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.127892 32493 tablet_service.cc:1468] Processing CreateTablet for tablet 16331ba6a34942a5a1b4ff4ac7bac55e (DEFAULT_TABLE table=default.Table_4533668b8daa4ff3ad00ccc95e97e032 [id=08d03b1b488c4b52abebcef8b3d0f853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.128716 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16331ba6a34942a5a1b4ff4ac7bac55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.130873 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affa3da7540648d28c5f00a3cf348eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.134479 339 ts_tablet_manager.cc:1397] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.095s user 0.004s sys 0.001s
I20250623 08:04:33.134578 848 raft_consensus.cc:491] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.134817 848 raft_consensus.cc:513] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.135663 848 leader_election.cc:290] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.137691 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 92005abc61dd40ef9da8124cadc29b6a) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:33.138386 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 (DEFAULT_TABLE table=default.Table_5d6ec4f6805b4e0883b8d0419625d033 [id=545b478879aa4f4cb35393e3065d3ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.138803 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.147300 32488 tablet_service.cc:1468] Processing CreateTablet for tablet bb06a340f82047ac8ec3516edb08f4b6 (DEFAULT_TABLE table=default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.147449 892 webserver.cc:469] Webserver started at http://127.31.69.61:44043/ using document root <none> and password file <none>
I20250623 08:04:33.147825 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb06a340f82047ac8ec3516edb08f4b6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:33.148612 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 43831.7us
I20250623 08:04:33.148706 892 fs_manager.cc:362] Metadata directory not provided
I20250623 08:04:33.148834 892 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250623 08:04:33.149205 32314 consensus_peers.cc:487] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b980c746a3c042bfae90aa076acbb1df. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:33.149787 1118 tablet.cc:1722] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:33.155325 339 raft_consensus.cc:357] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.156594 1118 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:35542 (request call id 40) took 1934 ms. Trace:
I20250623 08:04:33.156752 1118 rpcz_store.cc:276] 0623 08:04:31.221639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.221739 (+ 100us) service_pool.cc:225] Handling call
0623 08:04:33.156527 (+1934788us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:31.880931 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:31.881081 (+ 150us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:32.382523 (+501442us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:32.382534 (+ 11us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:32.382616 (+ 82us) alter_schema_op.cc:128] START. Timestamp: 1750665872382557
0623 08:04:32.382619 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:32.383152 (+ 533us) log.cc:844] Serialized 173 byte log entry
0623 08:04:33.149646 (+766494us) op_driver.cc:464] REPLICATION: finished
0623 08:04:33.149729 (+ 83us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:33.150152 (+ 423us) log.cc:844] Serialized 5 byte log entry
0623 08:04:33.155475 (+ 5323us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.156445 (+ 970us) log.cc:844] Serialized 14 byte log entry
0623 08:04:33.156483 (+ 38us) alter_schema_op.cc:73] Released schema lock
0623 08:04:33.156489 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":32,"prepare.queue_time_us":622169,"prepare.run_cpu_time_us":16526,"prepare.run_wall_time_us":502927,"replication_time_us":766989}]]}
I20250623 08:04:33.156332 339 raft_consensus.cc:383] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.158090 339 raft_consensus.cc:738] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.158496 339 consensus_queue.cc:260] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.160844 339 ts_tablet_manager.cc:1428] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250623 08:04:33.163153 979 ts_tablet_manager.cc:1918] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:33.163568 892 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.004s sys 0.006s
I20250623 08:04:33.164291 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "205da880685b49b3b8669f73dc2c563b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.165093 32314 leader_election.cc:304] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:33.165681 1174 raft_consensus.cc:2747] T 432d1642c0e44119b5bc05858463a136 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:33.168046 32314 leader_election.cc:304] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:33.168255 339 tablet_bootstrap.cc:492] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:33.168417 1174 raft_consensus.cc:2747] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.178576 1212 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:33.168478 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 60) took 1452 ms. Trace:
I20250623 08:04:33.179405 981 rpcz_store.cc:276] 0623 08:04:31.715518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.716166 (+ 648us) service_pool.cc:225] Handling call
0623 08:04:33.063056 (+1346890us) ts_tablet_manager.cc:1145] Deleting tablet 38ecd4c243b840f4b66d8e01078dca64
0623 08:04:33.165950 (+102894us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.168418 (+ 2468us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346562}
I20250623 08:04:33.179589 32483 tablet_service.cc:1468] Processing CreateTablet for tablet 74da35cd34c14284af004001b25aa531 (DEFAULT_TABLE table=default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.180199 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 63) took 1464 ms. Trace:
I20250623 08:04:33.180308 981 rpcz_store.cc:276] 0623 08:04:31.715867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.716608 (+ 741us) service_pool.cc:225] Handling call
0623 08:04:33.180012 (+1463404us) ts_tablet_manager.cc:1145] Deleting tablet 6926c49b787941d98c647543bd3b9ed5
0623 08:04:33.180023 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.180127 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1463219}
I20250623 08:04:33.180435 892 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.000s sys 0.008s
W20250623 08:04:33.165709 32589 leader_election.cc:343] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.175961 32631 tablet_service.cc:1468] Processing CreateTablet for tablet db81250d1e164a7d94affd764fc0ff0c (DEFAULT_TABLE table=default.Table_c2afde03c007450390a0466fe1d39344 [id=77e1411772b54c89a9a8435654249287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.180612 892 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data,/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
uuid: "9752155c20ee42fbb7ff7b7d5b682444"
format_stamp: "Formatted at 2025-06-23 08:04:18 on dist-test-slave-bxhn"
I20250623 08:04:33.181532 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "358cb6ac24414bdfbc32155e732f9ac8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:33.181779 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db81250d1e164a7d94affd764fc0ff0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.181592 892 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250623 08:04:33.182032 32589 consensus_peers.cc:487] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:33.182503 979 log.cc:1199] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/04eda92e14144733ab71a577dd3bd2b0
I20250623 08:04:33.183014 979 ts_tablet_manager.cc:1939] T 04eda92e14144733ab71a577dd3bd2b0 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:33.183812 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 67) took 2172 ms. Trace:
I20250623 08:04:33.183907 979 rpcz_store.cc:276] 0623 08:04:31.011474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.011518 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:32.940343 (+1928825us) ts_tablet_manager.cc:1145] Deleting tablet 04eda92e14144733ab71a577dd3bd2b0
0623 08:04:32.940354 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.043489 (+103135us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.163028 (+119539us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.183755 (+ 20727us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1928638}
I20250623 08:04:33.187188 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74da35cd34c14284af004001b25aa531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.187664 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 64) took 1471 ms. Trace:
I20250623 08:04:33.188040 981 rpcz_store.cc:276] 0623 08:04:31.716075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.716807 (+ 732us) service_pool.cc:225] Handling call
0623 08:04:33.182629 (+1465822us) ts_tablet_manager.cc:1145] Deleting tablet 52d0a356c74f4daea9093c998b4bf73e
0623 08:04:33.182640 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.187598 (+ 4958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465612}
I20250623 08:04:33.188486 343 tablet_bootstrap.cc:654] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.191038 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 61) took 1475 ms. Trace:
I20250623 08:04:33.191143 981 rpcz_store.cc:276] 0623 08:04:31.715623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.718081 (+ 2458us) service_pool.cc:225] Handling call
0623 08:04:33.190841 (+1472760us) ts_tablet_manager.cc:1145] Deleting tablet 70f9d21b8c144f9cada9bdbe0fbb1538
0623 08:04:33.190851 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.190977 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472485}
I20250623 08:04:33.194648 342 tablet_bootstrap.cc:654] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.161486 952 ts_tablet_manager.cc:1918] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:33.172099 32489 tablet_service.cc:1468] Processing CreateTablet for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (DEFAULT_TABLE table=default.Table_a7cbd31e5c664fba885a13dcbd9105ab [id=86224d949cab4304b8bcd6b429415163]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.196197 952 log.cc:1199] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/2060a643d0da47efa868f52bdb331b1f
I20250623 08:04:33.196623 952 ts_tablet_manager.cc:1939] T 2060a643d0da47efa868f52bdb331b1f P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:33.196892 339 tablet_bootstrap.cc:654] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.197494 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 34) took 2241 ms. Trace:
I20250623 08:04:33.196645 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f217a8eb12014fcf89b9df4a2cddf5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.200239 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 62) took 1484 ms. Trace:
I20250623 08:04:33.200834 981 rpcz_store.cc:276] 0623 08:04:31.715765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.717054 (+ 1289us) service_pool.cc:225] Handling call
0623 08:04:33.200038 (+1482984us) ts_tablet_manager.cc:1145] Deleting tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f
0623 08:04:33.200047 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.200161 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1467603}
I20250623 08:04:33.201256 981 tablet_replica.cc:331] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:33.201536 981 raft_consensus.cc:2241] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.201787 981 pending_rounds.cc:70] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:33.201925 981 pending_rounds.cc:77] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7170727407953252352 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "5fdc9e53da264a5ca1f70e1846c906f0" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_0242542a86ba40a28474c9832cd8ed3c" dest_uuid: "59f7c614254f42009306b13337f06576" new_extra_config { } }
I20250623 08:04:33.202545 981 raft_consensus.cc:2270] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.206208 32314 leader_election.cc:304] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:33.206552 610 raft_consensus.cc:2747] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.184368 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 04eda92e14144733ab71a577dd3bd2b0 (table default.Table_f4a87ef61f514d7eb4e6ebfb76458e7a [id=89a3fc1cb42f4902a69ea52945cedc0c]) successfully deleted
I20250623 08:04:33.181777 32655 raft_consensus.cc:3058] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.207996 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet f3c850021e914530bb106ab7166be7c9 (table default.Table_fd49e04fcac14f0194cc9fed910601fc [id=17b055700da64cc1b19f93dd5edf40d5]) successfully deleted
W20250623 08:04:33.208432 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet f4a0b9402a264800a1ea39707c10da0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4a0b9402a264800a1ea39707c10da0c already in progress: creating tablet
W20250623 08:04:33.208779 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet d6ed4cb252b1452087f8ee53ec78d985 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6ed4cb252b1452087f8ee53ec78d985 already in progress: creating tablet
I20250623 08:04:33.211521 32655 raft_consensus.cc:2466] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:33.189098 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0773fc60d4a463596f536f15d58bf57" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:33.212218 32654 raft_consensus.cc:2466] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:33.188838 1136 raft_consensus.cc:513] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.214105 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdec6e9573b041e0a57d1e0e0239d97f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:33.214781 32589 leader_election.cc:343] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.215041 1136 leader_election.cc:290] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:33.215063 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet f3d32798168345edb9898751d083057c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3d32798168345edb9898751d083057c already in progress: creating tablet
I20250623 08:04:33.215423 32058 catalog_manager.cc:5582] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:33.178637 32490 tablet_service.cc:1468] Processing CreateTablet for tablet db81250d1e164a7d94affd764fc0ff0c (DEFAULT_TABLE table=default.Table_c2afde03c007450390a0466fe1d39344 [id=77e1411772b54c89a9a8435654249287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.217283 1121 tablet.cc:1722] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:33.217639 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db81250d1e164a7d94affd764fc0ff0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.218917 848 consensus_queue.cc:1035] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:33.220618 979 tablet_replica.cc:331] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
W20250623 08:04:33.224905 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 38ecd4c243b840f4b66d8e01078dca64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38ecd4c243b840f4b66d8e01078dca64 already in progress: creating tablet
I20250623 08:04:33.225898 981 ts_tablet_manager.cc:1905] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:33.226306 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 6926c49b787941d98c647543bd3b9ed5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6926c49b787941d98c647543bd3b9ed5 already in progress: creating tablet
W20250623 08:04:33.226819 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 52d0a356c74f4daea9093c998b4bf73e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52d0a356c74f4daea9093c998b4bf73e already in progress: creating tablet
I20250623 08:04:33.227809 1136 raft_consensus.cc:491] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.228039 1136 raft_consensus.cc:513] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.229832 1136 leader_election.cc:290] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:33.230545 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70f9d21b8c144f9cada9bdbe0fbb1538 already in progress: creating tablet
W20250623 08:04:33.231160 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f already in progress: creating tablet
I20250623 08:04:33.229663 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31ecc74317d449e5b557b07a4fd14895" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:33.232383 32589 leader_election.cc:343] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.234814 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "965f1f0a107b4653b6aac01e0d8e259a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.234427 979 raft_consensus.cc:2241] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:33.235059 1118 tablet.cc:1722] T 2414d24ab6c54f36bb1e554f7231a5d7 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:33.235250 979 raft_consensus.cc:2270] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.235872 979 ts_tablet_manager.cc:1905] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.238049 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3d32798168345edb9898751d083057c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.240069 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "358cb6ac24414bdfbc32155e732f9ac8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.244074 1118 tablet.cc:1722] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:33.249727 892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 08:04:33.250144 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "358cb6ac24414bdfbc32155e732f9ac8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:33.250924 892 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 08:04:33.251269 892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 08:04:33.255134 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0773fc60d4a463596f536f15d58bf57" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.256287 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 803d3607d75443568b7ddc9f0c0bdc90 (DEFAULT_TABLE table=default.Table_98f40c99d9ff4819b958cd84cc5dc3a9 [id=13cf327a200c490992163a70f5a275cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.256719 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803d3607d75443568b7ddc9f0c0bdc90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.259189 32361 tablet_service.cc:1468] Processing CreateTablet for tablet d19d468ce13d4108a9badc36fc503306 (DEFAULT_TABLE table=default.Table_5c43b8626a6640e4a61e495ab2ea5697 [id=46664e505d9648ca889484123253a613]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.259742 979 ts_tablet_manager.cc:1918] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.259927 979 log.cc:1199] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/f0117cbb09dd4067836266c9d6a32adf
I20250623 08:04:33.260519 979 ts_tablet_manager.cc:1939] T f0117cbb09dd4067836266c9d6a32adf P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:33.256304 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 90037c523a1f4118af268e088bb97062 (DEFAULT_TABLE table=default.Table_5cf87169ed0d4987b396195d518aa055 [id=5edc67ced51e4cde9dbfcf448d24b87d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.261500 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 68) took 2249 ms. Trace:
I20250623 08:04:33.261631 979 rpcz_store.cc:276] 0623 08:04:31.011832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.011873 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:33.220518 (+2208645us) ts_tablet_manager.cc:1145] Deleting tablet f0117cbb09dd4067836266c9d6a32adf
0623 08:04:33.220527 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.246244 (+ 25717us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.249278 (+ 3034us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.261461 (+ 12183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2208490}
I20250623 08:04:33.261999 979 tablet_replica.cc:331] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:33.262231 979 raft_consensus.cc:2241] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.262442 979 raft_consensus.cc:2270] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.262389 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet f0117cbb09dd4067836266c9d6a32adf (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:33.262977 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d19d468ce13d4108a9badc36fc503306. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.263088 979 ts_tablet_manager.cc:1905] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.264489 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 (DEFAULT_TABLE table=default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.266201 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 74da35cd34c14284af004001b25aa531 (DEFAULT_TABLE table=default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.266503 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 16331ba6a34942a5a1b4ff4ac7bac55e (DEFAULT_TABLE table=default.Table_4533668b8daa4ff3ad00ccc95e97e032 [id=08d03b1b488c4b52abebcef8b3d0f853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.266698 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90037c523a1f4118af268e088bb97062. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.266883 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 5443f12974bf41929d9cd937f146cd56 (DEFAULT_TABLE table=default.Table_eda413af2cf84c009c8fc4b375b57e2a [id=898ec01859bf4ba29e38b4eaf3d5392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.267951 32357 tablet_service.cc:1468] Processing CreateTablet for tablet affa3da7540648d28c5f00a3cf348eb1 (DEFAULT_TABLE table=default.Table_9c6bc2fbfc274d2e8414205f629cb439 [id=42f65c762e2b46469d3ff189b698b447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.268380 32349 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 92005abc61dd40ef9da8124cadc29b6a) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:33.269874 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16331ba6a34942a5a1b4ff4ac7bac55e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.269161 32358 tablet_service.cc:1468] Processing CreateTablet for tablet bb06a340f82047ac8ec3516edb08f4b6 (DEFAULT_TABLE table=default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.270800 32349 tablet_service.cc:1468] Processing CreateTablet for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 (DEFAULT_TABLE table=default.Table_5d6ec4f6805b4e0883b8d0419625d033 [id=545b478879aa4f4cb35393e3065d3ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.273485 32354 tablet_service.cc:1468] Processing CreateTablet for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (DEFAULT_TABLE table=default.Table_a7cbd31e5c664fba885a13dcbd9105ab [id=86224d949cab4304b8bcd6b429415163]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.276312 32360 tablet_service.cc:1468] Processing CreateTablet for tablet a0e9b2f810014e81a29ca9b41cd3f8ba (DEFAULT_TABLE table=default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.276670 32348 tablet_service.cc:1468] Processing CreateTablet for tablet db81250d1e164a7d94affd764fc0ff0c (DEFAULT_TABLE table=default.Table_c2afde03c007450390a0466fe1d39344 [id=77e1411772b54c89a9a8435654249287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.276965 610 raft_consensus.cc:491] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.277818 610 raft_consensus.cc:513] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.278347 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb06a340f82047ac8ec3516edb08f4b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.279088 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f217a8eb12014fcf89b9df4a2cddf5f6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:33.278931 32314 consensus_peers.cc:487] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2795cb14e6db4116a4c9ee539bd4e82d. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:33.279402 610 leader_election.cc:290] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.279654 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db81250d1e164a7d94affd764fc0ff0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.281670 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e9b2f810014e81a29ca9b41cd3f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.281919 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:33.282186 32654 raft_consensus.cc:2391] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:33.283099 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affa3da7540648d28c5f00a3cf348eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.284933 981 ts_tablet_manager.cc:1918] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:33.285074 981 log.cc:1199] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/5fdc9e53da264a5ca1f70e1846c906f0
I20250623 08:04:33.289801 979 ts_tablet_manager.cc:1918] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:33.290071 979 log.cc:1199] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/16d6b1e81e4d438f85d8fee4dd7d8b7d
I20250623 08:04:33.290644 979 ts_tablet_manager.cc:1939] T 16d6b1e81e4d438f85d8fee4dd7d8b7d P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:33.291560 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 69) took 2279 ms. Trace:
I20250623 08:04:33.291752 979 rpcz_store.cc:276] 0623 08:04:31.012150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.012185 (+ 35us) service_pool.cc:225] Handling call
0623 08:04:33.261932 (+2249747us) ts_tablet_manager.cc:1145] Deleting tablet 16d6b1e81e4d438f85d8fee4dd7d8b7d
0623 08:04:33.261941 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.271223 (+ 9282us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.289614 (+ 18391us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.291502 (+ 1888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2249549}
I20250623 08:04:33.293085 979 tablet_replica.cc:331] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:33.294482 979 raft_consensus.cc:2241] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.296017 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74da35cd34c14284af004001b25aa531. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.197597 952 rpcz_store.cc:276] 0623 08:04:30.955972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.956327 (+ 355us) service_pool.cc:225] Handling call
0623 08:04:32.780127 (+1823800us) ts_tablet_manager.cc:1145] Deleting tablet 2060a643d0da47efa868f52bdb331b1f
0623 08:04:32.780140 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:32.967175 (+187035us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.161347 (+194172us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.197438 (+ 36091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1822782}
I20250623 08:04:33.312882 32453 leader_election.cc:304] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:33.314157 1071 tablet.cc:1722] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:33.318660 1159 raft_consensus.cc:2802] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:33.326602 979 raft_consensus.cc:2270] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.326807 1159 raft_consensus.cc:491] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:33.326948 1159 raft_consensus.cc:3058] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.328639 979 ts_tablet_manager.cc:1905] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.330703 1159 raft_consensus.cc:513] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.334326 981 ts_tablet_manager.cc:1939] T 5fdc9e53da264a5ca1f70e1846c906f0 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.335292 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 43) took 1622 ms. Trace:
I20250623 08:04:33.335449 981 rpcz_store.cc:276] 0623 08:04:31.712907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.769557 (+ 56650us) service_pool.cc:225] Handling call
0623 08:04:33.201173 (+1431616us) ts_tablet_manager.cc:1145] Deleting tablet 5fdc9e53da264a5ca1f70e1846c906f0
0623 08:04:33.201181 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.251613 (+ 50432us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.283352 (+ 31739us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.335241 (+ 51889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337468}
I20250623 08:04:33.335999 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 5fdc9e53da264a5ca1f70e1846c906f0 (table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]) successfully deleted
I20250623 08:04:33.336251 981 tablet_replica.cc:331] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:33.336508 981 raft_consensus.cc:2241] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.336695 981 raft_consensus.cc:2270] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:33.336738 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9c8d480b8ba49a1be140b0e0a79f4e4 already in progress: creating tablet
W20250623 08:04:33.336987 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet ea702abb31fb41498ab61bbb05b1f7e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea702abb31fb41498ab61bbb05b1f7e2 already in progress: creating tablet
I20250623 08:04:33.332865 1071 tablet.cc:1722] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:33.337520 32349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.338109 1159 leader_election.cc:290] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.339686 746 raft_consensus.cc:491] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.340718 979 ts_tablet_manager.cc:1918] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.340931 979 log.cc:1199] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/d4221d7500d1458db75926d7b274a654
I20250623 08:04:33.341442 979 ts_tablet_manager.cc:1939] T d4221d7500d1458db75926d7b274a654 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:33.342037 339 tablet_bootstrap.cc:492] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:33.343575 746 raft_consensus.cc:513] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.343789 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 70) took 2331 ms. Trace:
I20250623 08:04:33.343900 979 rpcz_store.cc:276] 0623 08:04:31.012529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.012571 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:33.292500 (+2279929us) ts_tablet_manager.cc:1145] Deleting tablet d4221d7500d1458db75926d7b274a654
0623 08:04:33.292510 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.333534 (+ 41024us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.340275 (+ 6741us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.343755 (+ 3480us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2279736}
I20250623 08:04:33.344266 746 leader_election.cc:290] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.344367 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet d4221d7500d1458db75926d7b274a654 (table default.Table_110816be0fae488f9cc26b302495009c [id=6de8bb3add89407bb0d3e5685ed5b6bf]) successfully deleted
W20250623 08:04:33.345211 32314 leader_election.cc:343] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: c0f87fd62f1545089d00757f8214593a
I20250623 08:04:33.345551 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.345815 32515 raft_consensus.cc:2391] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:33.346793 343 tablet_bootstrap.cc:492] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:33.346951 343 ts_tablet_manager.cc:1397] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.229s user 0.008s sys 0.005s
I20250623 08:04:33.349277 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfa787f828b411ab68d81bb896f7fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.354523 979 tablet_replica.cc:331] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:33.355794 979 raft_consensus.cc:2241] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:33.356262 979 raft_consensus.cc:2270] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.357074 979 ts_tablet_manager.cc:1905] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.361306 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5443f12974bf41929d9cd937f146cd56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.355432 343 raft_consensus.cc:357] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.361400 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:33.361516 343 raft_consensus.cc:383] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.361639 343 raft_consensus.cc:738] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.361653 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:33.361923 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f02f36d6c1174a23a1f1043a431d0b6b (table default.Table_75451c4863a34537a328617c53d46cae [id=be88c6ecf20d4ab1b3757c27ce998f87]). Expected version 2 got 0
I20250623 08:04:33.362116 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:33.362241 343 consensus_queue.cc:260] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.362660 32054 catalog_manager.cc:5582] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:33.363090 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:33.363126 342 tablet_bootstrap.cc:492] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.363269 342 ts_tablet_manager.cc:1397] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.248s user 0.016s sys 0.009s
I20250623 08:04:33.364012 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ba3d2fa01d574dd992dc904ac024ba3a (table default.Table_d496fd69b6f140b88b5abc86c8839ee2 [id=6c57db68f5ed45a3b64d1e3a02ce8a2a]). Expected version 1 got 0
I20250623 08:04:33.364045 342 raft_consensus.cc:357] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.364215 342 raft_consensus.cc:383] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.364292 342 raft_consensus.cc:738] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.364287 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 31a67b70cbc64bb3bfd5a34865109a01 (table default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e [id=7a569a9fe3bd4c608ddca48a734df1da]). Expected version 2 got 1
I20250623 08:04:33.364495 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:33.364674 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_3af33890e8ff40c881af74580254633b [id=8b20894a91c54a3eb08d5e1b4a42d269]). Expected version 3 got 0
I20250623 08:04:33.364910 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:33.365103 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (table default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b]). Expected version 1 got 0
I20250623 08:04:33.366384 952 tablet_replica.cc:331] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:33.366528 343 ts_tablet_manager.cc:1428] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250623 08:04:33.369068 343 tablet_bootstrap.cc:492] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:33.371591 952 raft_consensus.cc:2241] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.371763 32054 catalog_manager.cc:5582] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:33.372223 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]). Expected version 2 got 0
I20250623 08:04:33.372588 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:33.372905 952 raft_consensus.cc:2270] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.373193 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:33.373461 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_bc4ef67b97864fb99c405ffe92e2d006 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 1 got 0
W20250623 08:04:33.371945 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 91957.4us
W20250623 08:04:33.374132 32453 consensus_peers.cc:487] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35ab9f77a81a4490abb0d2c0c69cd90f. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:33.375219 952 ts_tablet_manager.cc:1905] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.378036 339 ts_tablet_manager.cc:1397] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.174s user 0.002s sys 0.003s
I20250623 08:04:33.375656 32453 leader_election.cc:304] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:33.379012 1159 raft_consensus.cc:2802] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:33.378964 339 raft_consensus.cc:357] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.379077 32453 leader_election.cc:304] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:33.379166 339 raft_consensus.cc:383] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.379232 339 raft_consensus.cc:738] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.379317 343 tablet_bootstrap.cc:654] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.379324 848 raft_consensus.cc:2802] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:33.379411 848 raft_consensus.cc:491] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:33.379519 848 raft_consensus.cc:3058] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.379448 339 consensus_queue.cc:260] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.379981 339 ts_tablet_manager.cc:1428] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:33.380326 339 tablet_bootstrap.cc:492] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:33.380851 32314 leader_election.cc:304] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:33.380965 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 2060a643d0da47efa868f52bdb331b1f (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]) successfully deleted
I20250623 08:04:33.381701 746 raft_consensus.cc:3058] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.381948 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3d32798168345edb9898751d083057c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:33.382149 32654 raft_consensus.cc:3058] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.382118 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdec6e9573b041e0a57d1e0e0239d97f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:33.382386 32517 raft_consensus.cc:3058] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:33.383713 32314 leader_election.cc:343] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: c0f87fd62f1545089d00757f8214593a
I20250623 08:04:33.383862 32314 leader_election.cc:304] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:33.383944 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3d32798168345edb9898751d083057c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:33.384232 981 ts_tablet_manager.cc:1905] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.385110 32654 raft_consensus.cc:2466] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:33.385133 32517 raft_consensus.cc:2466] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:33.385466 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31ecc74317d449e5b557b07a4fd14895" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.386253 32374 raft_consensus.cc:3058] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.387421 1054 raft_consensus.cc:491] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.387565 1054 raft_consensus.cc:513] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.388183 1054 leader_election.cc:290] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.387463 343 tablet_bootstrap.cc:492] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:33.388500 343 ts_tablet_manager.cc:1397] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
W20250623 08:04:33.388527 32453 consensus_peers.cc:487] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:33.388909 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:33.389492 343 raft_consensus.cc:357] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:33.389750 32453 leader_election.cc:343] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.389889 32453 leader_election.cc:304] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.391932 1054 raft_consensus.cc:2747] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.392531 343 raft_consensus.cc:383] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.393354 848 raft_consensus.cc:513] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.394218 343 raft_consensus.cc:738] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.394615 343 consensus_queue.cc:260] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.395177 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0861107ab4624a05bd7a4d24b420c502 (table default.Table_977fe8fc665f4267b394065e22f67eea [id=be9754ecb40a413d9940224adbd7e2eb]). Expected version 2 got 0
I20250623 08:04:33.396418 343 ts_tablet_manager.cc:1428] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250623 08:04:33.397209 848 leader_election.cc:290] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.397974 343 tablet_bootstrap.cc:492] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:33.399783 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 3 got 0
I20250623 08:04:33.394030 979 ts_tablet_manager.cc:1918] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.400256 979 log.cc:1199] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/c6a97b34e386443fa19a9601621b5017
I20250623 08:04:33.401018 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b84b8b8713774ab4896842feb9fb075e (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:33.401372 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:33.401755 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:33.402199 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_087d39ad116a4ac2a1cf1050403ef202 [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 1 got 0
I20250623 08:04:33.402508 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:33.402683 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_32db9f68dd974fb8aff372a8eeccce3d [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 1 got 0
I20250623 08:04:33.402945 979 ts_tablet_manager.cc:1939] T c6a97b34e386443fa19a9601621b5017 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:33.402952 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:33.403106 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f83f68ebda7e4826bbb17278cd4ac5a8 (table default.Table_6acb69304c424830a635f22f12343b31 [id=c1782341d0d34662889cff00d557ff81]). Expected version 1 got 0
I20250623 08:04:33.403443 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:33.403818 32054 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 18ea63263b9b466e922169eecafab747 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]). Expected version 1 got 0
I20250623 08:04:33.404072 1159 raft_consensus.cc:695] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:33.404201 32056 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f279b63b55c64e548ded27b02025c9f5
I20250623 08:04:33.404664 1159 consensus_queue.cc:237] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.404870 32056 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 179) took 1153 ms. Trace:
I20250623 08:04:33.405251 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet f279b63b55c64e548ded27b02025c9f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:33.400360 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "205da880685b49b3b8669f73dc2c563b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:33.394634 32453 leader_election.cc:343] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.404987 32056 rpcz_store.cc:276] 0623 08:04:32.251422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.498815 (+247393us) service_pool.cc:225] Handling call
0623 08:04:32.611846 (+113031us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fa306 00007f19c7644956 00007f19c7644685 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:32.612167 (+ 321us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:32.612293 (+ 126us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:32.612348 (+ 55us) catalog_manager.cc:2799] Locking tablets
0623 08:04:32.612486 (+ 138us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:32.719258 (+106772us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:32.719320 (+ 62us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:33.404119 (+684799us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:33.404819 (+ 700us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:32.637984 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:32.638071 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:32.638076 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:32.638078 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:32.638161 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:32.638169 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:32.638174 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:32.638267 (+ 93us) tablet.cc:699] Row locks acquired
0623 08:04:32.638269 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:32.638294 (+ 25us) write_op.cc:270] Start()
0623 08:04:32.638317 (+ 23us) write_op.cc:276] Timestamp: P: 1750665872638233 usec, L: 0
0623 08:04:32.638320 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:32.638450 (+ 130us) log.cc:844] Serialized 725 byte log entry
0623 08:04:32.716998 (+ 78548us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.717540 (+ 542us) write_op.cc:301] APPLY: starting
0623 08:04:32.717555 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:32.717581 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:32.717604 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:32.717658 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:32.717676 (+ 18us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:32.717683 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:32.717733 (+ 50us) log.cc:844] Serialized 29 byte log entry
0623 08:04:32.717791 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.717805 (+ 14us) write_op.cc:454] Released schema lock
0623 08:04:32.717810 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":259373824,"child_traces":[["op",{"apply.queue_time_us":506,"apply.run_cpu_time_us":326,"apply.run_wall_time_us":326,"num_ops":2,"prepare.queue_time_us":24883,"prepare.run_cpu_time_us":615,"prepare.run_wall_time_us":57952,"raft.queue_time_us":57110,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":78665,"spinlock_wait_cycles":10880,"thread_start_us":28462,"threads_started":1}]]}
I20250623 08:04:33.406008 32056 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_087d39ad116a4ac2a1cf1050403ef202" } new_table_name: "default.Table_049497d4290c4bdfaf1e609b8b2ac9ac" modify_external_catalogs: true
W20250623 08:04:33.406706 32453 leader_election.cc:343] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.406776 1175 raft_consensus.cc:2747] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:33.406967 32059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87648b11b10b45fc9cc8c998bbceae92 consensus state has no leader
W20250623 08:04:33.407075 32453 leader_election.cc:343] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.407209 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 358cb6ac24414bdfbc32155e732f9ac8
W20250623 08:04:33.407344 32453 leader_election.cc:343] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.407747 32057 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 169) took 1247 ms. Trace:
I20250623 08:04:33.408193 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 358cb6ac24414bdfbc32155e732f9ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:33.407852 32057 rpcz_store.cc:276] 0623 08:04:32.159949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.159997 (+ 48us) service_pool.cc:225] Handling call
0623 08:04:32.211385 (+ 51388us) spinlock_profiling.cc:243] Waited 51.2 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fa306 00007f19c7644956 00007f19c7644685 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:32.211790 (+ 405us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:32.212149 (+ 359us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:32.212213 (+ 64us) catalog_manager.cc:2799] Locking tablets
0623 08:04:32.212361 (+ 148us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:32.610068 (+397707us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:32.610150 (+ 82us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:33.407144 (+796994us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:33.407708 (+ 564us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:32.478718 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:32.478822 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:32.478827 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:32.478829 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:32.478916 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:32.478927 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:32.478934 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:32.479022 (+ 88us) tablet.cc:699] Row locks acquired
0623 08:04:32.479024 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:32.479050 (+ 26us) write_op.cc:270] Start()
0623 08:04:32.479074 (+ 24us) write_op.cc:276] Timestamp: P: 1750665872479006 usec, L: 0
0623 08:04:32.479077 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:32.479215 (+ 138us) log.cc:844] Serialized 756 byte log entry
0623 08:04:32.609191 (+129976us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.609286 (+ 95us) write_op.cc:301] APPLY: starting
0623 08:04:32.609302 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:32.609350 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:32.609420 (+ 70us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:32.609547 (+ 127us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:32.609585 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:32.609592 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:32.609724 (+ 132us) log.cc:844] Serialized 29 byte log entry
0623 08:04:32.609818 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.609842 (+ 24us) write_op.cc:454] Released schema lock
0623 08:04:32.609849 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":117686528,"child_traces":[["op",{"apply.queue_time_us":40,"apply.run_cpu_time_us":628,"apply.run_wall_time_us":628,"num_ops":2,"prepare.queue_time_us":265728,"prepare.run_cpu_time_us":569,"prepare.run_wall_time_us":643,"raft.queue_time_us":45,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":130085}]]}
I20250623 08:04:33.408516 32057 master_service.cc:948] ReplaceTablet: received request to replace tablet dc0dad440b494018bc0d6cfd40940bc1 from {username='slave'} at 127.0.0.1:36116
W20250623 08:04:33.408842 32050 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b405f87bb1e4d0686a1f627835d901f consensus state has no leader
W20250623 08:04:33.408915 32453 leader_election.cc:343] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:33.409065 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 73453eb1a0394cfe8d49adcc71d4f3e9 consensus state has no leader
I20250623 08:04:33.410215 32589 leader_election.cc:304] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:33.410653 32589 leader_election.cc:343] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.410799 32589 leader_election.cc:304] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.411269 1128 raft_consensus.cc:2802] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:33.411404 1128 raft_consensus.cc:695] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:33.411679 1128 consensus_queue.cc:237] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.411808 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20e7a78cfc384033810603316b9e66bf" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:33.411981 32654 raft_consensus.cc:2466] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:33.412647 1011 raft_consensus.cc:2747] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.413209 952 ts_tablet_manager.cc:1918] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.413340 952 log.cc:1199] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/d4221d7500d1458db75926d7b274a654
I20250623 08:04:33.414203 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0773fc60d4a463596f536f15d58bf57" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:33.414393 32654 raft_consensus.cc:3058] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.417336 32654 raft_consensus.cc:2466] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:33.418051 32589 leader_election.cc:343] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.418179 32589 leader_election.cc:304] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.418444 1136 raft_consensus.cc:2747] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.419080 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet f279b63b55c64e548ded27b02025c9f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:33.419117 32051 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 175) took 1228 ms. Trace:
I20250623 08:04:33.419660 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet f279b63b55c64e548ded27b02025c9f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:33.419687 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 358cb6ac24414bdfbc32155e732f9ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:33.419205 32051 rpcz_store.cc:276] 0623 08:04:32.191017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.216177 (+ 25160us) service_pool.cc:225] Handling call
0623 08:04:32.452536 (+236359us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fb5a6 00007f19c7646616 00007f19c7646345 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:32.452802 (+ 266us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:32.453006 (+ 204us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:32.453188 (+ 182us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:32.453247 (+ 59us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:32.717923 (+264676us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:33.419061 (+701138us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:32.636145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:32.636231 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:32.636236 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:32.636238 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:32.636313 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:32.636320 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:32.636324 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:32.636371 (+ 47us) tablet.cc:699] Row locks acquired
0623 08:04:32.636373 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:32.636400 (+ 27us) write_op.cc:270] Start()
0623 08:04:32.636427 (+ 27us) write_op.cc:276] Timestamp: P: 1750665872636339 usec, L: 0
0623 08:04:32.636433 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:32.636581 (+ 148us) log.cc:844] Serialized 405 byte log entry
0623 08:04:32.716911 (+ 80330us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.717153 (+ 242us) write_op.cc:301] APPLY: starting
0623 08:04:32.717166 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:32.717193 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:32.717217 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:32.717269 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:32.717291 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:32.717296 (+ 5us) write_op.cc:312] APPLY: finished
0623 08:04:32.717336 (+ 40us) log.cc:844] Serialized 23 byte log entry
0623 08:04:32.717363 (+ 27us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.717384 (+ 21us) write_op.cc:454] Released schema lock
0623 08:04:32.717390 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:32.717459 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.717463 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":543031040,"child_traces":[["op",{"apply.queue_time_us":203,"apply.run_cpu_time_us":298,"apply.run_wall_time_us":297,"mutex_wait_us":1887,"num_ops":1,"prepare.queue_time_us":182471,"prepare.run_cpu_time_us":539,"prepare.run_wall_time_us":906,"raft.queue_time_us":928,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":80460,"spinlock_wait_cycles":896}]]}
I20250623 08:04:33.419799 32453 leader_election.cc:304] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.420106 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fae6ac5a256346149e8cf56ae9dbdf3b" } new_table_name: "default.Table_b8a2a531e35748bab1f7751ece1f6b06" modify_external_catalogs: true
I20250623 08:04:33.420912 952 ts_tablet_manager.cc:1939] T d4221d7500d1458db75926d7b274a654 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:33.421509 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20e7a78cfc384033810603316b9e66bf" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.421922 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0773fc60d4a463596f536f15d58bf57" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:33.421211 1153 raft_consensus.cc:2802] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:33.422906 32059 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 174) took 1244 ms. Trace:
I20250623 08:04:33.423022 1153 raft_consensus.cc:695] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:33.423316 1153 consensus_queue.cc:237] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.422993 32059 rpcz_store.cc:276] 0623 08:04:32.178536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.204356 (+ 25820us) service_pool.cc:225] Handling call
0623 08:04:32.207035 (+ 2679us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fb5a6 00007f19c7646616 00007f19c7646345 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:32.207349 (+ 314us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:32.207595 (+ 246us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:32.207809 (+ 214us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:32.207905 (+ 96us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:32.497835 (+289930us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:33.422848 (+925013us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:32.477551 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:32.477640 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:32.477645 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:32.477648 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:32.477731 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:32.477739 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:32.477743 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:32.477794 (+ 51us) tablet.cc:699] Row locks acquired
0623 08:04:32.477796 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:32.477822 (+ 26us) write_op.cc:270] Start()
0623 08:04:32.477849 (+ 27us) write_op.cc:276] Timestamp: P: 1750665872477778 usec, L: 0
0623 08:04:32.477853 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:32.477977 (+ 124us) log.cc:844] Serialized 405 byte log entry
0623 08:04:32.496604 (+ 18627us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.497475 (+ 871us) write_op.cc:301] APPLY: starting
0623 08:04:32.497485 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:32.497515 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:32.497545 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:32.497580 (+ 35us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:32.497606 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:32.497613 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:32.497665 (+ 52us) log.cc:844] Serialized 23 byte log entry
0623 08:04:32.497724 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.497755 (+ 31us) write_op.cc:454] Released schema lock
0623 08:04:32.497760 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:32.497820 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.497823 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":5579136,"child_traces":[["op",{"apply.queue_time_us":839,"apply.run_cpu_time_us":339,"apply.run_wall_time_us":339,"num_ops":1,"prepare.queue_time_us":269168,"prepare.run_cpu_time_us":506,"prepare.run_wall_time_us":555,"raft.queue_time_us":30,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":18723}]]}
I20250623 08:04:33.424005 32059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_b7a92f8a00ba4931827eaee759bc8f91" } modify_external_catalogs: true
I20250623 08:04:33.424201 32059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_b7a92f8a00ba4931827eaee759bc8f91" } modify_external_catalogs: true
I20250623 08:04:33.424710 746 raft_consensus.cc:2747] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.425648 32050 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 185) took 1019 ms. Trace:
I20250623 08:04:33.425753 32050 rpcz_store.cc:276] 0623 08:04:32.405943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.855420 (+449477us) service_pool.cc:225] Handling call
0623 08:04:32.880843 (+ 25423us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fb5a6 00007f19c7646616 00007f19c7646345 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:32.881572 (+ 729us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:32.881833 (+ 261us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:32.882023 (+ 190us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:32.882118 (+ 95us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:32.937529 (+ 55411us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:33.425610 (+488081us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:32.903199 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:32.903363 (+ 164us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:32.903372 (+ 9us) write_op.cc:435] Acquired schema lock
0623 08:04:32.903375 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:32.903516 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:32.903536 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:32.903545 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:32.903639 (+ 94us) tablet.cc:699] Row locks acquired
0623 08:04:32.903643 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:32.903694 (+ 51us) write_op.cc:270] Start()
0623 08:04:32.903736 (+ 42us) write_op.cc:276] Timestamp: P: 1750665872903651 usec, L: 0
0623 08:04:32.903742 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:32.903978 (+ 236us) log.cc:844] Serialized 405 byte log entry
0623 08:04:32.935915 (+ 31937us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.936302 (+ 387us) write_op.cc:301] APPLY: starting
0623 08:04:32.936322 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:32.936370 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:32.936409 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:32.936463 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:32.936501 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:32.936511 (+ 10us) write_op.cc:312] APPLY: finished
0623 08:04:32.936594 (+ 83us) log.cc:844] Serialized 23 byte log entry
0623 08:04:32.937310 (+ 716us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.937330 (+ 20us) write_op.cc:454] Released schema lock
0623 08:04:32.937340 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:32.937425 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.937429 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":57593216,"child_traces":[["op",{"apply.queue_time_us":318,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":1121,"num_ops":1,"prepare.queue_time_us":20239,"prepare.run_cpu_time_us":990,"prepare.run_wall_time_us":27536,"raft.queue_time_us":755,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":288,"replication_time_us":32140}]]}
I20250623 08:04:33.426497 32050 master_service.cc:948] ReplaceTablet: received request to replace tablet 485538f7c63948818f20a25c02381519 from {username='slave'} at 127.0.0.1:36116
W20250623 08:04:33.435060 32044 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 73453eb1a0394cfe8d49adcc71d4f3e9 consensus state has no leader
I20250623 08:04:33.435861 32453 leader_election.cc:304] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:33.437256 1054 raft_consensus.cc:2802] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:33.437356 1054 raft_consensus.cc:491] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:33.437469 1054 raft_consensus.cc:3058] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.440300 1054 raft_consensus.cc:513] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.441046 1054 leader_election.cc:290] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.436553 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 37) took 2478 ms. Trace:
I20250623 08:04:33.441694 952 rpcz_store.cc:276] 0623 08:04:30.958293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.958340 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:33.306875 (+2348535us) ts_tablet_manager.cc:1145] Deleting tablet d4221d7500d1458db75926d7b274a654
0623 08:04:33.306888 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.390413 (+ 83525us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.413076 (+ 22663us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.436500 (+ 23424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2348198}
I20250623 08:04:33.443045 952 tablet_replica.cc:331] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:33.443883 952 raft_consensus.cc:2241] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.447470 952 raft_consensus.cc:2270] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.448639 952 ts_tablet_manager.cc:1905] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.449669 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 9049e3d9b39f420bb51221808409fc54
I20250623 08:04:33.441538 32453 leader_election.cc:304] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:33.451066 1054 raft_consensus.cc:2802] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:33.451704 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 9049e3d9b39f420bb51221808409fc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:33.453163 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 9049e3d9b39f420bb51221808409fc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:33.453212 1054 raft_consensus.cc:695] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:33.454155 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20e7a78cfc384033810603316b9e66bf" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:33.454401 32654 raft_consensus.cc:3058] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.453469 32360 tablet_service.cc:1515] Processing DeleteTablet for tablet 358cb6ac24414bdfbc32155e732f9ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:33.457285 32654 raft_consensus.cc:2466] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:33.457995 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet d4221d7500d1458db75926d7b274a654 (table default.Table_110816be0fae488f9cc26b302495009c [id=6de8bb3add89407bb0d3e5685ed5b6bf]) successfully deleted
W20250623 08:04:33.458041 32056 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fd478c007914d2cb09d929b50322a64 consensus state has no leader
I20250623 08:04:33.460232 32056 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 190) took 1039 ms. Trace:
I20250623 08:04:33.460351 32056 rpcz_store.cc:276] 0623 08:04:32.420479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.405846 (+985367us) service_pool.cc:225] Handling call
0623 08:04:33.406208 (+ 362us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:33.406508 (+ 300us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:33.406736 (+ 228us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:33.418276 (+ 11540us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:33.457697 (+ 39421us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:33.460188 (+ 2491us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:33.428485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:33.428702 (+ 217us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:33.428713 (+ 11us) write_op.cc:435] Acquired schema lock
0623 08:04:33.428717 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:33.428853 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:33.428874 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:33.428882 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:33.428966 (+ 84us) tablet.cc:699] Row locks acquired
0623 08:04:33.428970 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:33.429011 (+ 41us) write_op.cc:270] Start()
0623 08:04:33.429052 (+ 41us) write_op.cc:276] Timestamp: P: 1750665873428970 usec, L: 0
0623 08:04:33.429058 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:33.429293 (+ 235us) log.cc:844] Serialized 405 byte log entry
0623 08:04:33.441022 (+ 11729us) op_driver.cc:464] REPLICATION: finished
0623 08:04:33.441507 (+ 485us) write_op.cc:301] APPLY: starting
0623 08:04:33.441529 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:33.441573 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:33.441606 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:33.441662 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:33.441702 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:33.441709 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:33.441807 (+ 98us) log.cc:844] Serialized 23 byte log entry
0623 08:04:33.441869 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.441888 (+ 19us) write_op.cc:454] Released schema lock
0623 08:04:33.441896 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:33.441966 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.441970 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":432,"apply.run_cpu_time_us":452,"apply.run_wall_time_us":452,"num_ops":1,"prepare.queue_time_us":9538,"prepare.run_cpu_time_us":956,"prepare.run_wall_time_us":957,"raft.queue_time_us":3250,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":371,"replication_time_us":11931}]]}
I20250623 08:04:33.463421 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20e7a78cfc384033810603316b9e66bf" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:33.469071 32589 consensus_peers.cc:487] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:33.453533 1054 consensus_queue.cc:237] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.470656 342 consensus_queue.cc:260] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.475975 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet 9049e3d9b39f420bb51221808409fc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:33.476174 339 tablet_bootstrap.cc:654] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.476205 32453 leader_election.cc:304] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:33.476498 848 raft_consensus.cc:2802] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:33.476687 848 raft_consensus.cc:695] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:33.477025 848 consensus_queue.cc:237] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:33.478710 32453 leader_election.cc:343] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.478734 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_cc3b284f368e4123bf1a2da98f4f4fa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:33.479100 32453 leader_election.cc:343] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:33.479481 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc3b284f368e4123bf1a2da98f4f4fa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:33.480587 32453 leader_election.cc:343] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.481032 32059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_b523bec7182443139e891357d295e3a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:33.481789 32059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b523bec7182443139e891357d295e3a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:33.482357 339 tablet_bootstrap.cc:492] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:33.482580 339 ts_tablet_manager.cc:1397] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.000s
I20250623 08:04:33.489833 339 raft_consensus.cc:357] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.490032 339 raft_consensus.cc:383] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.490134 339 raft_consensus.cc:738] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.490394 339 consensus_queue.cc:260] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.490939 339 ts_tablet_manager.cc:1428] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250623 08:04:33.492357 339 tablet_bootstrap.cc:492] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:33.492645 32314 consensus_peers.cc:487] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4221d7500d1458db75926d7b274a654. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:33.492854 343 tablet_bootstrap.cc:654] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.494556 339 tablet_bootstrap.cc:654] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.497423 342 ts_tablet_manager.cc:1428] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.133s user 0.003s sys 0.000s
I20250623 08:04:33.498610 32050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 485538f7c63948818f20a25c02381519
I20250623 08:04:33.499240 32050 catalog_manager.cc:6652] ReplaceTablet: tablet 485538f7c63948818f20a25c02381519 deleted and replaced by tablet 6857437f85ab40f6a24e74ce3b5efbdc
I20250623 08:04:33.499711 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_ea4d110c161f4b008acde5f22ffa3562"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:33.500325 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea4d110c161f4b008acde5f22ffa3562 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:33.505623 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 485538f7c63948818f20a25c02381519 with delete_type TABLET_DATA_DELETED (replaced by tablet 6857437f85ab40f6a24e74ce3b5efbdc) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:33.506249 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet dc0dad440b494018bc0d6cfd40940bc1
I20250623 08:04:33.506810 32057 catalog_manager.cc:6652] ReplaceTablet: tablet dc0dad440b494018bc0d6cfd40940bc1 deleted and replaced by tablet 8f83a6d453ec4ffe85282b4385df5f11
I20250623 08:04:33.509605 981 ts_tablet_manager.cc:1918] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:33.515205 32056 master_service.cc:948] ReplaceTablet: received request to replace tablet effd4fff43fe4d4fb36458986e1dfce1 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:33.517606 981 log.cc:1199] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/2060a643d0da47efa868f52bdb331b1f
I20250623 08:04:33.518201 981 ts_tablet_manager.cc:1939] T 2060a643d0da47efa868f52bdb331b1f P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.519560 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_e541bb700f4c4e2abc268babf435c6b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:33.520666 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 69) took 1140 ms. Trace:
W20250623 08:04:33.522276 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e541bb700f4c4e2abc268babf435c6b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:33.524411 32059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3ef2e86e99164cce8e882e6954e2ec17" } modify_external_catalogs: true
I20250623 08:04:33.524713 32059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3ef2e86e99164cce8e882e6954e2ec17" } modify_external_catalogs: true
I20250623 08:04:33.527530 32493 tablet_service.cc:1515] Processing DeleteTablet for tablet dc0dad440b494018bc0d6cfd40940bc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f83a6d453ec4ffe85282b4385df5f11) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:33.527802 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 485538f7c63948818f20a25c02381519 with delete_type TABLET_DATA_DELETED (replaced by tablet 6857437f85ab40f6a24e74ce3b5efbdc) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:33.528272 981 rpcz_store.cc:276] 0623 08:04:32.380519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.380569 (+ 50us) service_pool.cc:225] Handling call
0623 08:04:33.336192 (+955623us) ts_tablet_manager.cc:1145] Deleting tablet 2060a643d0da47efa868f52bdb331b1f
0623 08:04:33.336198 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.471395 (+135197us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.509450 (+ 38055us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.520595 (+ 11145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":955358}
I20250623 08:04:33.530813 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet dc0dad440b494018bc0d6cfd40940bc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f83a6d453ec4ffe85282b4385df5f11) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:33.531955 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 68) took 1151 ms. Trace:
I20250623 08:04:33.532269 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 2060a643d0da47efa868f52bdb331b1f (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]) successfully deleted
I20250623 08:04:33.532653 981 rpcz_store.cc:276] 0623 08:04:32.380408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.380891 (+ 483us) service_pool.cc:225] Handling call
0623 08:04:33.531770 (+1150879us) ts_tablet_manager.cc:1145] Deleting tablet 2ecf4af9e476495683bd63cac20d4672
0623 08:04:33.531783 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.531900 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1150605}
I20250623 08:04:33.533175 981 tablet_replica.cc:331] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
W20250623 08:04:33.533545 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 2ecf4af9e476495683bd63cac20d4672 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ecf4af9e476495683bd63cac20d4672 already in progress: creating tablet
I20250623 08:04:33.534137 981 raft_consensus.cc:2241] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:33.534675 981 raft_consensus.cc:2270] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.535853 981 ts_tablet_manager.cc:1905] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.493332 1128 raft_consensus.cc:491] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.543354 1128 raft_consensus.cc:513] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.544059 1128 leader_election.cc:290] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:33.501801 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 485538f7c63948818f20a25c02381519 with delete_type TABLET_DATA_DELETED (replaced by tablet 6857437f85ab40f6a24e74ce3b5efbdc) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:33.510176 1136 consensus_queue.cc:1035] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:33.513836 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet dc0dad440b494018bc0d6cfd40940bc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f83a6d453ec4ffe85282b4385df5f11) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:33.514071 944 raft_consensus.cc:491] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.544687 944 raft_consensus.cc:513] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.544919 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025365d2fa0a43dd983d395a05cc2af1" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.545266 944 leader_election.cc:290] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:33.541534 1011 raft_consensus.cc:491] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.545469 1011 raft_consensus.cc:513] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.545703 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffafd10f26740c9a5b6c9bdc3bb23a0" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.546140 1011 leader_election.cc:290] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:33.546317 32589 leader_election.cc:343] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.547016 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9c8d480b8ba49a1be140b0e0a79f4e4" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:33.547675 32589 leader_election.cc:343] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.547781 342 tablet_bootstrap.cc:492] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:33.548331 32589 leader_election.cc:343] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.552249 981 ts_tablet_manager.cc:1918] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.552450 981 log.cc:1199] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/279624b8bb834b95a4f08045968ebfb4
I20250623 08:04:33.552700 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 71) took 2477 ms. Trace:
I20250623 08:04:33.552867 979 rpcz_store.cc:276] 0623 08:04:31.013949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.013994 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:33.352574 (+2338580us) ts_tablet_manager.cc:1145] Deleting tablet c6a97b34e386443fa19a9601621b5017
0623 08:04:33.352590 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.373706 (+ 21116us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.393919 (+ 20213us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.491294 (+ 97375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2338370}
I20250623 08:04:33.553292 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet c6a97b34e386443fa19a9601621b5017 (table default.Table_a1a27d7e7fb04ab3bc9aca9948a43bca [id=c330ef9f7f0549a7abc1eeab676ec11d]) successfully deleted
I20250623 08:04:33.553046 981 ts_tablet_manager.cc:1939] T 279624b8bb834b95a4f08045968ebfb4 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.554348 979 tablet_replica.cc:331] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:33.554419 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 71) took 1163 ms. Trace:
I20250623 08:04:33.554961 981 rpcz_store.cc:276] 0623 08:04:32.391139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.391184 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:33.533078 (+1141894us) ts_tablet_manager.cc:1145] Deleting tablet 279624b8bb834b95a4f08045968ebfb4
0623 08:04:33.533089 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.540859 (+ 7770us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.552082 (+ 11223us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.554376 (+ 2294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141628}
I20250623 08:04:33.555549 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 279624b8bb834b95a4f08045968ebfb4 (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]) successfully deleted
I20250623 08:04:33.556414 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 70) took 1165 ms. Trace:
I20250623 08:04:33.560242 981 rpcz_store.cc:276] 0623 08:04:32.390959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.391484 (+ 525us) service_pool.cc:225] Handling call
0623 08:04:33.556177 (+1164693us) ts_tablet_manager.cc:1145] Deleting tablet 4cb66c7a273a412db3ae77af9e30d036
0623 08:04:33.556191 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.556368 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164466}
W20250623 08:04:33.561177 32314 consensus_peers.cc:487] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:33.561444 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 4cb66c7a273a412db3ae77af9e30d036 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cb66c7a273a412db3ae77af9e30d036 already in progress: creating tablet
I20250623 08:04:33.562247 981 tablet_replica.cc:331] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:33.562942 981 raft_consensus.cc:2241] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.563166 981 raft_consensus.cc:2270] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.563491 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025365d2fa0a43dd983d395a05cc2af1" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.564342 981 ts_tablet_manager.cc:1905] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.566454 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:33.567237 342 tablet_bootstrap.cc:654] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.576107 981 ts_tablet_manager.cc:1918] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.576416 981 log.cc:1199] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/1e47b10f46b8402e94354ebca573c194
I20250623 08:04:33.576989 981 ts_tablet_manager.cc:1939] T 1e47b10f46b8402e94354ebca573c194 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.577437 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 73) took 1180 ms. Trace:
I20250623 08:04:33.577608 981 rpcz_store.cc:276] 0623 08:04:32.396937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.396983 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:33.562179 (+1165196us) ts_tablet_manager.cc:1145] Deleting tablet 1e47b10f46b8402e94354ebca573c194
0623 08:04:33.562190 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.568630 (+ 6440us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.575954 (+ 7324us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.577382 (+ 1428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164949}
I20250623 08:04:33.578966 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 1e47b10f46b8402e94354ebca573c194 (table default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]) successfully deleted
I20250623 08:04:33.579874 981 tablet_replica.cc:331] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:33.580202 981 raft_consensus.cc:2241] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:33.580718 981 raft_consensus.cc:2270] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.581192 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_3d6c2aca7a944301bc67f7901cfef8b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:33.581910 981 ts_tablet_manager.cc:1905] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.582453 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_d3115e0a02304f6d80807841b76504e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:33.583076 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d6c2aca7a944301bc67f7901cfef8b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:33.583076 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3115e0a02304f6d80807841b76504e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:33.585937 32056 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet effd4fff43fe4d4fb36458986e1dfce1
I20250623 08:04:33.586498 32056 catalog_manager.cc:6652] ReplaceTablet: tablet effd4fff43fe4d4fb36458986e1dfce1 deleted and replaced by tablet d5cd316e6f4447be958611ab07ed31fc
I20250623 08:04:33.587131 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet effd4fff43fe4d4fb36458986e1dfce1 with delete_type TABLET_DATA_DELETED (replaced by tablet d5cd316e6f4447be958611ab07ed31fc) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:33.587373 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_1476eae1111b4f1dbd7c0f5dd0dd9b34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:33.588151 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1476eae1111b4f1dbd7c0f5dd0dd9b34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:33.589841 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffafd10f26740c9a5b6c9bdc3bb23a0" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.592646 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9c8d480b8ba49a1be140b0e0a79f4e4" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.592914 981 ts_tablet_manager.cc:1918] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.593026 981 log.cc:1199] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/19536c16ff32460e9db571e4ce411bab
I20250623 08:04:33.594168 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet effd4fff43fe4d4fb36458986e1dfce1 with delete_type TABLET_DATA_DELETED (replaced by tablet d5cd316e6f4447be958611ab07ed31fc) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:33.594797 981 ts_tablet_manager.cc:1939] T 19536c16ff32460e9db571e4ce411bab P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.595229 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 72) took 1198 ms. Trace:
I20250623 08:04:33.595386 981 rpcz_store.cc:276] 0623 08:04:32.396835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.397262 (+ 427us) service_pool.cc:225] Handling call
0623 08:04:33.579759 (+1182497us) ts_tablet_manager.cc:1145] Deleting tablet 19536c16ff32460e9db571e4ce411bab
0623 08:04:33.579774 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.586248 (+ 6474us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.592782 (+ 6534us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.595193 (+ 2411us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1182245}
I20250623 08:04:33.595882 981 tablet_replica.cc:331] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:33.595985 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 19536c16ff32460e9db571e4ce411bab (table default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030 [id=f8961c82f9384b0db128a791e00351f6]) successfully deleted
I20250623 08:04:33.596264 981 raft_consensus.cc:2241] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.596623 981 raft_consensus.cc:2270] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.597345 981 ts_tablet_manager.cc:1905] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.605823 981 ts_tablet_manager.cc:1918] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.606098 981 log.cc:1199] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/5629d11c44e241d598f3e519b2fe1a44
I20250623 08:04:33.606634 981 ts_tablet_manager.cc:1939] T 5629d11c44e241d598f3e519b2fe1a44 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.607568 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 75) took 1197 ms. Trace:
I20250623 08:04:33.607745 981 rpcz_store.cc:276] 0623 08:04:32.410223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.410580 (+ 357us) service_pool.cc:225] Handling call
0623 08:04:33.595795 (+1185215us) ts_tablet_manager.cc:1145] Deleting tablet 5629d11c44e241d598f3e519b2fe1a44
0623 08:04:33.595807 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.602108 (+ 6301us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.605709 (+ 3601us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.607501 (+ 1792us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1184776}
I20250623 08:04:33.608260 981 tablet_replica.cc:331] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:33.608398 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 5629d11c44e241d598f3e519b2fe1a44 (table default.Table_4d4535cab44545839eb3a559385e395a [id=bb849e86d7084959830c68207fb99061]) successfully deleted
I20250623 08:04:33.608601 981 raft_consensus.cc:2241] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.608847 981 raft_consensus.cc:2270] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.609477 981 ts_tablet_manager.cc:1905] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.610530 952 ts_tablet_manager.cc:1918] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.610783 952 log.cc:1199] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/2795cb14e6db4116a4c9ee539bd4e82d
I20250623 08:04:33.611282 952 ts_tablet_manager.cc:1939] T 2795cb14e6db4116a4c9ee539bd4e82d P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:33.611690 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 38) took 2652 ms. Trace:
I20250623 08:04:33.611847 952 rpcz_store.cc:276] 0623 08:04:30.958799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.958842 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:33.442967 (+2484125us) ts_tablet_manager.cc:1145] Deleting tablet 2795cb14e6db4116a4c9ee539bd4e82d
0623 08:04:33.442977 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.606536 (+163559us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.610423 (+ 3887us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.611626 (+ 1203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2483840}
I20250623 08:04:33.612303 952 tablet_replica.cc:331] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:33.612767 952 raft_consensus.cc:2241] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:33.613027 952 raft_consensus.cc:2270] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.613950 952 ts_tablet_manager.cc:1905] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.616277 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 2795cb14e6db4116a4c9ee539bd4e82d (table default.Table_cd24de3acd89483480238a058a33e72c [id=9b918ea1a6ca412d9db93bbb1215732f]) successfully deleted
I20250623 08:04:33.616564 981 ts_tablet_manager.cc:1918] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.616765 981 log.cc:1199] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/175ee90133994154b7e4b58b0770db3b
I20250623 08:04:33.616834 342 tablet_bootstrap.cc:492] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.617026 342 ts_tablet_manager.cc:1397] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.118s user 0.004s sys 0.000s
I20250623 08:04:33.617360 981 ts_tablet_manager.cc:1939] T 175ee90133994154b7e4b58b0770db3b P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.617931 342 raft_consensus.cc:357] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.618144 342 raft_consensus.cc:383] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.618240 342 raft_consensus.cc:738] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.618388 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 74) took 1219 ms. Trace:
I20250623 08:04:33.618505 342 consensus_queue.cc:260] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.618543 981 rpcz_store.cc:276] 0623 08:04:32.410014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.411061 (+ 1047us) service_pool.cc:225] Handling call
0623 08:04:33.608170 (+1197109us) ts_tablet_manager.cc:1145] Deleting tablet 175ee90133994154b7e4b58b0770db3b
0623 08:04:33.608180 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.613219 (+ 5039us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.616443 (+ 3224us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.618264 (+ 1821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196864}
I20250623 08:04:33.618960 342 ts_tablet_manager.cc:1428] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250623 08:04:33.619233 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 175ee90133994154b7e4b58b0770db3b (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]) successfully deleted
I20250623 08:04:33.619495 342 tablet_bootstrap.cc:492] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.619673 981 tablet_replica.cc:331] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:33.619910 981 raft_consensus.cc:2241] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:33.620296 981 raft_consensus.cc:2270] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.620918 981 ts_tablet_manager.cc:1905] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.621294 342 tablet_bootstrap.cc:654] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.627645 981 ts_tablet_manager.cc:1918] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.627846 981 log.cc:1199] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/2795cb14e6db4116a4c9ee539bd4e82d
I20250623 08:04:33.628319 981 ts_tablet_manager.cc:1939] T 2795cb14e6db4116a4c9ee539bd4e82d P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.629106 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 77) took 1217 ms. Trace:
I20250623 08:04:33.629256 981 rpcz_store.cc:276] 0623 08:04:32.411625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.411673 (+ 48us) service_pool.cc:225] Handling call
0623 08:04:33.619598 (+1207925us) ts_tablet_manager.cc:1145] Deleting tablet 2795cb14e6db4116a4c9ee539bd4e82d
0623 08:04:33.619609 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.624490 (+ 4881us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.627528 (+ 3038us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.629049 (+ 1521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1207623}
I20250623 08:04:33.629774 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 2795cb14e6db4116a4c9ee539bd4e82d (table default.Table_cd24de3acd89483480238a058a33e72c [id=9b918ea1a6ca412d9db93bbb1215732f]) successfully deleted
I20250623 08:04:33.630172 981 tablet_replica.cc:331] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:33.630427 981 raft_consensus.cc:2241] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:33.630847 981 raft_consensus.cc:2270] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.631582 981 ts_tablet_manager.cc:1905] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.638321 342 tablet_bootstrap.cc:492] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.638499 342 ts_tablet_manager.cc:1397] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250623 08:04:33.639423 342 raft_consensus.cc:357] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.639626 342 raft_consensus.cc:383] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.639737 342 raft_consensus.cc:738] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.640020 342 consensus_queue.cc:260] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.640573 342 ts_tablet_manager.cc:1428] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:33.587859 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet effd4fff43fe4d4fb36458986e1dfce1 with delete_type TABLET_DATA_DELETED (replaced by tablet d5cd316e6f4447be958611ab07ed31fc) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:33.589237 32589 leader_election.cc:343] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.632697 1011 raft_consensus.cc:491] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.640890 32589 leader_election.cc:304] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.640962 1011 raft_consensus.cc:513] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.641080 979 raft_consensus.cc:2241] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:33.641319 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 52075.1us
W20250623 08:04:33.641587 32589 leader_election.cc:343] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.641669 1011 leader_election.cc:290] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:33.641772 32589 leader_election.cc:304] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.633811 944 raft_consensus.cc:491] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.641954 944 raft_consensus.cc:513] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.642066 1128 raft_consensus.cc:2747] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:33.642177 32589 leader_election.cc:343] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.642299 32589 leader_election.cc:304] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:33.642529 944 leader_election.cc:290] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:33.642661 1128 raft_consensus.cc:2747] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.642879 1128 consensus_queue.cc:1035] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:33.643631 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1c82e4d2db64026a04c5c84bc1e09ec" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:33.643966 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "136e482995a24c5daec55a701f3091fa" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:33.644498 32589 leader_election.cc:343] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:33.644733 32589 leader_election.cc:343] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.641814 1011 raft_consensus.cc:2747] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.645835 979 raft_consensus.cc:2270] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.646395 979 ts_tablet_manager.cc:1905] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.652598 342 tablet_bootstrap.cc:492] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.653308 343 tablet_bootstrap.cc:492] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:33.653502 343 ts_tablet_manager.cc:1397] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.256s user 0.003s sys 0.000s
I20250623 08:04:33.654417 342 tablet_bootstrap.cc:654] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.654487 343 raft_consensus.cc:357] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.654693 343 raft_consensus.cc:383] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.654819 343 raft_consensus.cc:738] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.655057 343 consensus_queue.cc:260] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.655508 343 ts_tablet_manager.cc:1428] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:33.656558 343 tablet_bootstrap.cc:492] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:33.659260 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1c82e4d2db64026a04c5c84bc1e09ec" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.660036 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "136e482995a24c5daec55a701f3091fa" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.662848 32377 raft_consensus.cc:3058] T f44fd3c67185484e93450682364dc8a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:33.663501 32589 leader_election.cc:343] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.663690 32589 leader_election.cc:304] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:33.664170 32589 leader_election.cc:343] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.664309 32589 leader_election.cc:304] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:33.700407 32045 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.61:33313, user_credentials={real_user=slave}} blocked reactor thread for 127025us
I20250623 08:04:33.718865 1054 raft_consensus.cc:491] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:33.719179 1054 raft_consensus.cc:513] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.721455 1054 leader_election.cc:290] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.723225 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48da1e4759bb4fea87a45ad8a428b6e7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:33.723481 32655 raft_consensus.cc:2466] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:33.727247 32453 leader_election.cc:304] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:33.727679 1054 raft_consensus.cc:2802] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:33.727813 1054 raft_consensus.cc:491] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:33.727942 1054 raft_consensus.cc:3058] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.666517 944 raft_consensus.cc:2747] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.729454 1011 raft_consensus.cc:2747] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:33.729615 944 consensus_queue.cc:1035] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:33.730437 343 tablet_bootstrap.cc:654] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:33.737717 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 69456.5us
I20250623 08:04:33.739152 981 ts_tablet_manager.cc:1918] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.741328 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_5c6bbeaad8274e05904301c82a72aa74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:33.741945 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c6bbeaad8274e05904301c82a72aa74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:33.745220 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_ba8a245f5a564519b522c175b4117f52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:33.745779 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba8a245f5a564519b522c175b4117f52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:33.746863 1054 raft_consensus.cc:513] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.747539 1054 leader_election.cc:290] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:33.751042 32490 tablet_service.cc:1468] Processing CreateTablet for tablet f97f14cbaffb46d681d8b11c0213cae4 (DEFAULT_TABLE table=default.Table_dd7b43110df2433597689f77b73907a1 [id=27942e98cc084670bc96433efb754c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.751492 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48da1e4759bb4fea87a45ad8a428b6e7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:33.751725 32654 raft_consensus.cc:3058] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:33.753827 952 ts_tablet_manager.cc:1918] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:33.753966 952 log.cc:1199] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/279624b8bb834b95a4f08045968ebfb4
I20250623 08:04:33.754483 952 ts_tablet_manager.cc:1939] T 279624b8bb834b95a4f08045968ebfb4 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:33.755697 32654 raft_consensus.cc:2466] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:33.756229 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:33.756459 343 tablet_bootstrap.cc:492] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:33.757369 32453 leader_election.cc:304] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:33.757722 981 log.cc:1199] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/d4221d7500d1458db75926d7b274a654
I20250623 08:04:33.759280 343 ts_tablet_manager.cc:1397] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.006s
I20250623 08:04:33.760174 343 raft_consensus.cc:357] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.760438 343 raft_consensus.cc:383] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.760533 343 raft_consensus.cc:738] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.760749 343 consensus_queue.cc:260] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.761261 343 ts_tablet_manager.cc:1428] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:33.761577 343 tablet_bootstrap.cc:492] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:33.762223 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48da1e4759bb4fea87a45ad8a428b6e7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:33.763118 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:33.763417 343 tablet_bootstrap.cc:654] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.763561 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:33.763753 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]). Expected version 2 got 0
I20250623 08:04:33.763753 32654 raft_consensus.cc:1273] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Refusing update from remote peer 7351aa31800245659edc9837e27daffd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 08:04:33.764132 32052 catalog_manager.cc:5582] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:33.764520 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:33.764645 981 ts_tablet_manager.cc:1939] T d4221d7500d1458db75926d7b274a654 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:33.764633 32348 tablet_service.cc:1468] Processing CreateTablet for tablet f97f14cbaffb46d681d8b11c0213cae4 (DEFAULT_TABLE table=default.Table_dd7b43110df2433597689f77b73907a1 [id=27942e98cc084670bc96433efb754c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.764787 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_3af33890e8ff40c881af74580254633b [id=8b20894a91c54a3eb08d5e1b4a42d269]). Expected version 3 got 2
I20250623 08:04:33.765051 32052 catalog_manager.cc:5582] T f44fd3c67185484e93450682364dc8a8 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:33.765527 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
W20250623 08:04:33.766208 32453 leader_election.cc:343] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.766403 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:33.766798 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
W20250623 08:04:33.767314 32314 consensus_peers.cc:487] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0ee5a4fe03848229524cc0105226d8c. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:33.767346 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f97f14cbaffb46d681d8b11c0213cae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.768791 1054 raft_consensus.cc:2802] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:33.769035 1054 raft_consensus.cc:695] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:33.769492 1054 consensus_queue.cc:237] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.773454 746 consensus_queue.cc:1035] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250623 08:04:33.771742 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_5785d39307cb42758ded4062307c55e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:33.775585 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5785d39307cb42758ded4062307c55e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:33.775954 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48da1e4759bb4fea87a45ad8a428b6e7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:33.778111 32453 leader_election.cc:343] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:33.772205 32056 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_04e85a0dbf974a05a94b9a9713fcb3f2" } modify_external_catalogs: true
I20250623 08:04:33.778491 32056 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_04e85a0dbf974a05a94b9a9713fcb3f2" } modify_external_catalogs: true
I20250623 08:04:33.779485 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 76) took 1363 ms. Trace:
I20250623 08:04:33.767031 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:33.780885 981 rpcz_store.cc:276] 0623 08:04:32.411486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.411969 (+ 483us) service_pool.cc:225] Handling call
0623 08:04:33.630078 (+1218109us) ts_tablet_manager.cc:1145] Deleting tablet d4221d7500d1458db75926d7b274a654
0623 08:04:33.630090 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.638133 (+ 8043us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.691223 (+ 53090us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.775131 (+ 83908us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1217857}
I20250623 08:04:33.783182 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet d4221d7500d1458db75926d7b274a654 (table default.Table_110816be0fae488f9cc26b302495009c [id=6de8bb3add89407bb0d3e5685ed5b6bf]) successfully deleted
I20250623 08:04:33.786437 32632 tablet_service.cc:1468] Processing CreateTablet for tablet f97f14cbaffb46d681d8b11c0213cae4 (DEFAULT_TABLE table=default.Table_dd7b43110df2433597689f77b73907a1 [id=27942e98cc084670bc96433efb754c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.786928 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f97f14cbaffb46d681d8b11c0213cae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.790441 32057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6648aba827f24ba29ee9d94ee7c96089" } modify_external_catalogs: true
I20250623 08:04:33.790674 32057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6648aba827f24ba29ee9d94ee7c96089" } modify_external_catalogs: true
I20250623 08:04:33.793022 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 3136ec837bea4619ae3995ceec9b24d6 (DEFAULT_TABLE table=default.Table_fb2522ad8b664fdc9ec779b46b807ca1 [id=d87e2b6427ac4078ac64e66511258f72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.793478 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3136ec837bea4619ae3995ceec9b24d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.793421 32620 tablet_service.cc:1468] Processing CreateTablet for tablet ea77b97018b74708bd6057bd2d65f90b (DEFAULT_TABLE table=default.Table_5338d405e4ed40589c691093043355d1 [id=b9246ed2ad6242739ae14a362b4af828]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.794397 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea77b97018b74708bd6057bd2d65f90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.795243 32483 tablet_service.cc:1468] Processing CreateTablet for tablet 344ebe1a14bc4d3389b0161ff102f05c (DEFAULT_TABLE table=default.Table_da6195995899479b879a3d41e0c07ed4 [id=b3b1a699a6b44ff09cf7b4f105e04794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.795348 32624 tablet_service.cc:1468] Processing CreateTablet for tablet 9891997e026448e39943ae13082bfe03 (DEFAULT_TABLE table=default.Table_6ca32a3252134e6587bca37cff71c969 [id=a595e975752c4d63b7b2726a8de5eb99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.795704 32624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9891997e026448e39943ae13082bfe03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.795967 32488 tablet_service.cc:1468] Processing CreateTablet for tablet 3136ec837bea4619ae3995ceec9b24d6 (DEFAULT_TABLE table=default.Table_fb2522ad8b664fdc9ec779b46b807ca1 [id=d87e2b6427ac4078ac64e66511258f72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.796330 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 11bb55088ab24b94af3e509810890247 (DEFAULT_TABLE table=default.Table_49e1ee7e0d024625988be24a823a443c [id=9a7a76e61e1b43508b6f6080f5163417]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.796527 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 11bb55088ab24b94af3e509810890247 (DEFAULT_TABLE table=default.Table_49e1ee7e0d024625988be24a823a443c [id=9a7a76e61e1b43508b6f6080f5163417]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.796772 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bb55088ab24b94af3e509810890247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.797070 32493 tablet_service.cc:1468] Processing CreateTablet for tablet ea77b97018b74708bd6057bd2d65f90b (DEFAULT_TABLE table=default.Table_5338d405e4ed40589c691093043355d1 [id=b9246ed2ad6242739ae14a362b4af828]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.797302 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 344ebe1a14bc4d3389b0161ff102f05c (DEFAULT_TABLE table=default.Table_da6195995899479b879a3d41e0c07ed4 [id=b3b1a699a6b44ff09cf7b4f105e04794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.797564 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 90ceb0f8d5c94c4e9427042d8a35e16c (DEFAULT_TABLE table=default.Table_a0e4ca05337b448ca84bafd20b982a81 [id=6fa96454486845848e54e86fd64a0017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.797677 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 8111e6117c1541ab8905d5a5a7e37292 (DEFAULT_TABLE table=default.Table_6b3f6aad5faf4f2a88573140691c6291 [id=8e322f09172b40a8ba5aea16337fb020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.797926 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 (DEFAULT_TABLE table=default.Table_28d324870b1049289c32fbfcf8c44ebf [id=cf51a9276ee0490c920bebfef8fbfaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.798043 32487 tablet_service.cc:1468] Processing CreateTablet for tablet 9891997e026448e39943ae13082bfe03 (DEFAULT_TABLE table=default.Table_6ca32a3252134e6587bca37cff71c969 [id=a595e975752c4d63b7b2726a8de5eb99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.798415 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 90ceb0f8d5c94c4e9427042d8a35e16c (DEFAULT_TABLE table=default.Table_a0e4ca05337b448ca84bafd20b982a81 [id=6fa96454486845848e54e86fd64a0017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.798529 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 (DEFAULT_TABLE table=default.Table_28d324870b1049289c32fbfcf8c44ebf [id=cf51a9276ee0490c920bebfef8fbfaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.798885 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 (DEFAULT_TABLE table=default.Table_90920cfec34b4621bae5567c93e73c89 [id=46e55c7114e745f0a404ba3b141ddb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.799054 32484 tablet_service.cc:1468] Processing CreateTablet for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 (DEFAULT_TABLE table=default.Table_90920cfec34b4621bae5567c93e73c89 [id=46e55c7114e745f0a404ba3b141ddb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.799566 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 8111e6117c1541ab8905d5a5a7e37292 (DEFAULT_TABLE table=default.Table_6b3f6aad5faf4f2a88573140691c6291 [id=8e322f09172b40a8ba5aea16337fb020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.799974 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344ebe1a14bc4d3389b0161ff102f05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.801167 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 6b04e38d3da44301a450fe1fb5e2ada6 (DEFAULT_TABLE table=default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.801707 342 tablet_bootstrap.cc:492] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.801759 32623 tablet_service.cc:1468] Processing CreateTablet for tablet 6b04e38d3da44301a450fe1fb5e2ada6 (DEFAULT_TABLE table=default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.801885 342 ts_tablet_manager.cc:1397] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.161s user 0.001s sys 0.002s
I20250623 08:04:33.802043 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8111e6117c1541ab8905d5a5a7e37292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.802873 342 raft_consensus.cc:357] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.803094 342 raft_consensus.cc:383] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.803177 342 raft_consensus.cc:738] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.803421 342 consensus_queue.cc:260] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.804327 342 ts_tablet_manager.cc:1428] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:33.804654 342 tablet_bootstrap.cc:492] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.806661 342 tablet_bootstrap.cc:654] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.806779 32623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b04e38d3da44301a450fe1fb5e2ada6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.809521 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.811559 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 92005abc61dd40ef9da8124cadc29b6a (DEFAULT_TABLE table=default.Table_04e85a0dbf974a05a94b9a9713fcb3f2 [id=7febcca3a06346738e869f73028a972e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.812296 32496 tablet_service.cc:1468] Processing CreateTablet for tablet cac83131e6854890b803b3d89ded7cf8 (DEFAULT_TABLE table=default.Table_849f3be400444579943f3f649824f4ea [id=f223e5f800b14f08b72352e85db1a9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.813081 32500 tablet_service.cc:1468] Processing CreateTablet for tablet b97f3019b893441bae18146fef094bd3 (DEFAULT_TABLE table=default.Table_4d6c00c8083e4821aa4a209e9cee8a91 [id=6e032739cfda445a8eb2188ecf94c816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.822988 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90ceb0f8d5c94c4e9427042d8a35e16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.824645 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 92005abc61dd40ef9da8124cadc29b6a (DEFAULT_TABLE table=default.Table_04e85a0dbf974a05a94b9a9713fcb3f2 [id=7febcca3a06346738e869f73028a972e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.825081 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89bb7ea31b4d4ce7a3c063853142fbb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.825037 944 consensus_queue.cc:1035] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20250623 08:04:33.829972 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92005abc61dd40ef9da8124cadc29b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.831463 342 tablet_bootstrap.cc:492] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.831714 342 ts_tablet_manager.cc:1397] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.001s
I20250623 08:04:33.833129 342 raft_consensus.cc:357] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.833403 342 raft_consensus.cc:383] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.833535 342 raft_consensus.cc:738] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.833907 342 consensus_queue.cc:260] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.835222 342 ts_tablet_manager.cc:1428] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:33.836808 342 tablet_bootstrap.cc:492] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.841142 32629 tablet_service.cc:1468] Processing CreateTablet for tablet b97f3019b893441bae18146fef094bd3 (DEFAULT_TABLE table=default.Table_4d6c00c8083e4821aa4a209e9cee8a91 [id=6e032739cfda445a8eb2188ecf94c816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.841567 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b97f3019b893441bae18146fef094bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.849784 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f (DEFAULT_TABLE table=default.Table_ff6e3cec929a433b8ac68152b9095dfc [id=ce7434eee7664a8686eb22f28df7f1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.850466 32495 tablet_service.cc:1468] Processing CreateTablet for tablet a34da529ffde4595a23fc6596b30e3e9 (DEFAULT_TABLE table=default.Table_969668212d4f422ca8f2f34266315161 [id=eb6d9c5745e649978f87c97627ace9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.851567 32498 tablet_service.cc:1468] Processing CreateTablet for tablet d0107b41d5c04a5c818f6792f2c18ce5 (DEFAULT_TABLE table=default.Table_91d48bbc63414bb4a1fb90f438f52eba [id=6eafeaf2ab324e3ebcf4ad4ad168cb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.855842 342 tablet_bootstrap.cc:654] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.870291 32627 tablet_service.cc:1468] Processing CreateTablet for tablet cac83131e6854890b803b3d89ded7cf8 (DEFAULT_TABLE table=default.Table_849f3be400444579943f3f649824f4ea [id=f223e5f800b14f08b72352e85db1a9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.872579 342 tablet_bootstrap.cc:492] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.872925 342 ts_tablet_manager.cc:1397] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.003s
I20250623 08:04:33.874660 342 raft_consensus.cc:357] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.875123 342 raft_consensus.cc:383] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.875262 342 raft_consensus.cc:738] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.875715 342 consensus_queue.cc:260] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.876767 342 ts_tablet_manager.cc:1428] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250623 08:04:33.877377 342 tablet_bootstrap.cc:492] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.880791 342 tablet_bootstrap.cc:654] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.886636 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac83131e6854890b803b3d89ded7cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.903896 342 tablet_bootstrap.cc:492] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.904278 342 ts_tablet_manager.cc:1397] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.016s
I20250623 08:04:33.906107 342 raft_consensus.cc:357] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.906533 342 raft_consensus.cc:383] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.906800 342 raft_consensus.cc:738] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.907850 342 consensus_queue.cc:260] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.909291 342 ts_tablet_manager.cc:1428] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250623 08:04:33.909981 342 tablet_bootstrap.cc:492] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.915241 342 tablet_bootstrap.cc:654] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.924865 342 tablet_bootstrap.cc:492] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.925277 342 ts_tablet_manager.cc:1397] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20250623 08:04:33.927049 342 raft_consensus.cc:357] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.927532 342 raft_consensus.cc:383] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.927671 342 raft_consensus.cc:738] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.928287 342 consensus_queue.cc:260] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.929445 342 ts_tablet_manager.cc:1428] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250623 08:04:33.930287 342 tablet_bootstrap.cc:492] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.934154 981 tablet_replica.cc:331] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:33.936105 342 tablet_bootstrap.cc:654] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.947793 342 tablet_bootstrap.cc:492] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.948086 342 ts_tablet_manager.cc:1397] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.003s
I20250623 08:04:33.948518 981 raft_consensus.cc:2241] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:33.948897 981 raft_consensus.cc:2270] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:33.951444 32349 tablet_service.cc:1468] Processing CreateTablet for tablet 3136ec837bea4619ae3995ceec9b24d6 (DEFAULT_TABLE table=default.Table_fb2522ad8b664fdc9ec779b46b807ca1 [id=d87e2b6427ac4078ac64e66511258f72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.953949 342 raft_consensus.cc:357] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.954350 342 raft_consensus.cc:383] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.954506 342 raft_consensus.cc:738] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.955057 342 consensus_queue.cc:260] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.956269 342 ts_tablet_manager.cc:1428] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250623 08:04:33.956895 342 tablet_bootstrap.cc:492] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.957362 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 344ebe1a14bc4d3389b0161ff102f05c (DEFAULT_TABLE table=default.Table_da6195995899479b879a3d41e0c07ed4 [id=b3b1a699a6b44ff09cf7b4f105e04794]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.959010 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 11bb55088ab24b94af3e509810890247 (DEFAULT_TABLE table=default.Table_49e1ee7e0d024625988be24a823a443c [id=9a7a76e61e1b43508b6f6080f5163417]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.959506 32349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3136ec837bea4619ae3995ceec9b24d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.960296 981 ts_tablet_manager.cc:1905] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:33.960525 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bb55088ab24b94af3e509810890247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.961247 342 tablet_bootstrap.cc:654] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.966840 32357 tablet_service.cc:1468] Processing CreateTablet for tablet ea77b97018b74708bd6057bd2d65f90b (DEFAULT_TABLE table=default.Table_5338d405e4ed40589c691093043355d1 [id=b9246ed2ad6242739ae14a362b4af828]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.967355 339 tablet_bootstrap.cc:492] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:33.968796 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea77b97018b74708bd6057bd2d65f90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.971451 342 tablet_bootstrap.cc:492] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:33.971585 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344ebe1a14bc4d3389b0161ff102f05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.971794 342 ts_tablet_manager.cc:1397] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20250623 08:04:33.972457 32360 tablet_service.cc:1468] Processing CreateTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 (DEFAULT_TABLE table=default.Table_28d324870b1049289c32fbfcf8c44ebf [id=cf51a9276ee0490c920bebfef8fbfaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.973371 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89bb7ea31b4d4ce7a3c063853142fbb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.973592 342 raft_consensus.cc:357] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.974033 342 raft_consensus.cc:383] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.974282 342 raft_consensus.cc:738] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.974912 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 9891997e026448e39943ae13082bfe03 (DEFAULT_TABLE table=default.Table_6ca32a3252134e6587bca37cff71c969 [id=a595e975752c4d63b7b2726a8de5eb99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.974785 342 consensus_queue.cc:260] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.975303 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9891997e026448e39943ae13082bfe03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.975906 342 ts_tablet_manager.cc:1428] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250623 08:04:33.976817 342 tablet_bootstrap.cc:492] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:33.977566 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 90ceb0f8d5c94c4e9427042d8a35e16c (DEFAULT_TABLE table=default.Table_a0e4ca05337b448ca84bafd20b982a81 [id=6fa96454486845848e54e86fd64a0017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.977977 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90ceb0f8d5c94c4e9427042d8a35e16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.978818 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 (DEFAULT_TABLE table=default.Table_90920cfec34b4621bae5567c93e73c89 [id=46e55c7114e745f0a404ba3b141ddb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.979967 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.981062 342 tablet_bootstrap.cc:654] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.982025 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 6b04e38d3da44301a450fe1fb5e2ada6 (DEFAULT_TABLE table=default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.983657 32356 tablet_service.cc:1468] Processing CreateTablet for tablet d0107b41d5c04a5c818f6792f2c18ce5 (DEFAULT_TABLE table=default.Table_91d48bbc63414bb4a1fb90f438f52eba [id=6eafeaf2ab324e3ebcf4ad4ad168cb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.988577 339 ts_tablet_manager.cc:1397] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.476s user 0.029s sys 0.053s
I20250623 08:04:33.988659 32355 tablet_service.cc:1468] Processing CreateTablet for tablet b97f3019b893441bae18146fef094bd3 (DEFAULT_TABLE table=default.Table_4d6c00c8083e4821aa4a209e9cee8a91 [id=6e032739cfda445a8eb2188ecf94c816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.988745 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b04e38d3da44301a450fe1fb5e2ada6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.989030 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b97f3019b893441bae18146fef094bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.989502 339 raft_consensus.cc:357] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.989727 339 raft_consensus.cc:383] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:33.989807 339 raft_consensus.cc:738] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:33.990057 339 consensus_queue.cc:260] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:33.990630 339 ts_tablet_manager.cc:1428] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:33.990996 339 tablet_bootstrap.cc:492] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:33.992653 32345 tablet_service.cc:1468] Processing CreateTablet for tablet a34da529ffde4595a23fc6596b30e3e9 (DEFAULT_TABLE table=default.Table_969668212d4f422ca8f2f34266315161 [id=eb6d9c5745e649978f87c97627ace9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.992877 339 tablet_bootstrap.cc:654] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:33.993041 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34da529ffde4595a23fc6596b30e3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:33.996222 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 92005abc61dd40ef9da8124cadc29b6a (DEFAULT_TABLE table=default.Table_04e85a0dbf974a05a94b9a9713fcb3f2 [id=7febcca3a06346738e869f73028a972e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.996467 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 8111e6117c1541ab8905d5a5a7e37292 (DEFAULT_TABLE table=default.Table_6b3f6aad5faf4f2a88573140691c6291 [id=8e322f09172b40a8ba5aea16337fb020]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:33.999797 32351 tablet_service.cc:1468] Processing CreateTablet for tablet cac83131e6854890b803b3d89ded7cf8 (DEFAULT_TABLE table=default.Table_849f3be400444579943f3f649824f4ea [id=f223e5f800b14f08b72352e85db1a9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.002485 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0107b41d5c04a5c818f6792f2c18ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.004459 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f (DEFAULT_TABLE table=default.Table_ff6e3cec929a433b8ac68152b9095dfc [id=ce7434eee7664a8686eb22f28df7f1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.009747 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.015728 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92005abc61dd40ef9da8124cadc29b6a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:34.020881 32453 consensus_peers.cc:487] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e47b10f46b8402e94354ebca573c194. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:34.022377 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f (DEFAULT_TABLE table=default.Table_ff6e3cec929a433b8ac68152b9095dfc [id=ce7434eee7664a8686eb22f28df7f1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.022886 342 tablet_bootstrap.cc:492] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:34.023861 32630 tablet_service.cc:1468] Processing CreateTablet for tablet a34da529ffde4595a23fc6596b30e3e9 (DEFAULT_TABLE table=default.Table_969668212d4f422ca8f2f34266315161 [id=eb6d9c5745e649978f87c97627ace9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.024962 343 tablet_bootstrap.cc:492] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.024956 944 raft_consensus.cc:491] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.024329 342 ts_tablet_manager.cc:1397] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.007s
I20250623 08:04:34.026300 343 ts_tablet_manager.cc:1397] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.264s user 0.009s sys 0.034s
I20250623 08:04:34.027530 32518 raft_consensus.cc:1273] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Refusing update from remote peer 7351aa31800245659edc9837e27daffd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 08:04:34.028682 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:34.029742 944 raft_consensus.cc:513] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.030417 944 leader_election.cc:290] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:34.030692 1275 raft_consensus.cc:491] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.032256 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdc9d47f2232403fa71e14a6fe6f33c3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:34.030849 1275 raft_consensus.cc:513] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.033351 342 raft_consensus.cc:357] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.033864 342 raft_consensus.cc:383] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.034127 342 raft_consensus.cc:738] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.033551 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:34.027015 32620 tablet_service.cc:1468] Processing CreateTablet for tablet d0107b41d5c04a5c818f6792f2c18ce5 (DEFAULT_TABLE table=default.Table_91d48bbc63414bb4a1fb90f438f52eba [id=6eafeaf2ab324e3ebcf4ad4ad168cb82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.035112 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.035018 342 consensus_queue.cc:260] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.035426 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:34.038306 342 ts_tablet_manager.cc:1428] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.011s user 0.003s sys 0.001s
I20250623 08:04:34.039235 342 tablet_bootstrap.cc:492] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:34.038451 32050 catalog_manager.cc:5582] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:34.043262 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f83f68ebda7e4826bbb17278cd4ac5a8 (table default.Table_6acb69304c424830a635f22f12343b31 [id=c1782341d0d34662889cff00d557ff81]). Expected version 1 got 0
I20250623 08:04:34.044912 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:34.046496 342 tablet_bootstrap.cc:654] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.047008 1251 raft_consensus.cc:491] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.047497 1251 raft_consensus.cc:513] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.048190 1251 leader_election.cc:290] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.048945 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34da529ffde4595a23fc6596b30e3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.051875 1251 raft_consensus.cc:491] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.052045 1251 raft_consensus.cc:513] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.052646 1251 leader_election.cc:290] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:34.053175 32589 leader_election.cc:343] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.059459 1251 raft_consensus.cc:491] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.059623 1251 raft_consensus.cc:513] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.059657 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 18ea63263b9b466e922169eecafab747 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]). Expected version 1 got 0
I20250623 08:04:34.060248 1251 leader_election.cc:290] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.061404 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:34.063762 343 raft_consensus.cc:357] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.064275 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:34.065331 343 raft_consensus.cc:383] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.065434 343 raft_consensus.cc:738] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.066288 343 consensus_queue.cc:260] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.066015 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:34.066871 343 ts_tablet_manager.cc:1428] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20250623 08:04:34.071728 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 3 got 0
I20250623 08:04:34.073293 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:34.075181 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:34.077426 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:34.078636 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 771ecfc41e2b4007a4d213cec5b128c6 (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:34.080966 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b84b8b8713774ab4896842feb9fb075e (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:34.082226 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_bc1251ce708a4467b5b1c71a1df307e0 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 1 got 0
I20250623 08:04:34.090389 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdc9d47f2232403fa71e14a6fe6f33c3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.088160 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9ea74d20c5f4eefbcb8508ed1774026" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.088776 343 tablet_bootstrap.cc:492] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:34.094518 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63029399994480ba7ccabcadc1ee5e0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.094700 32650 raft_consensus.cc:2466] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.095690 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac83131e6854890b803b3d89ded7cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.096872 32453 leader_election.cc:304] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.096974 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8111e6117c1541ab8905d5a5a7e37292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.098546 657 raft_consensus.cc:491] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.096997 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "205da880685b49b3b8669f73dc2c563b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.100888 32655 raft_consensus.cc:2466] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:34.101130 32589 leader_election.cc:343] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.098713 657 raft_consensus.cc:513] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.101998 32453 leader_election.cc:304] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.102244 657 leader_election.cc:290] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.102449 657 raft_consensus.cc:2802] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.102563 657 raft_consensus.cc:491] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.102665 657 raft_consensus.cc:3058] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.102954 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "205da880685b49b3b8669f73dc2c563b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.103118 32656 raft_consensus.cc:2466] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.105526 657 raft_consensus.cc:513] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.107705 32589 leader_election.cc:304] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:34.108014 32314 consensus_peers.cc:487] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d8f38a6de1e45e7bed4abdced684c92. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:34.113862 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63029399994480ba7ccabcadc1ee5e0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.115345 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9ea74d20c5f4eefbcb8508ed1774026" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.117321 1163 consensus_queue.cc:1035] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20250623 08:04:34.105823 848 raft_consensus.cc:2802] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.106967 32453 leader_election.cc:304] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.123790 848 raft_consensus.cc:491] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.123940 848 raft_consensus.cc:3058] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.124140 1159 raft_consensus.cc:2802] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.124251 1159 raft_consensus.cc:491] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.124336 1159 raft_consensus.cc:3058] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.127102 848 raft_consensus.cc:513] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:34.127280 32453 leader_election.cc:343] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:34.127578 32453 leader_election.cc:343] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.127681 848 leader_election.cc:290] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.128356 657 leader_election.cc:290] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.131145 1159 raft_consensus.cc:513] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.131688 1159 leader_election.cc:290] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.135985 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dc696236a15420e9666cca708068fb2" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.136526 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9ea74d20c5f4eefbcb8508ed1774026" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.142563 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dc696236a15420e9666cca708068fb2" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.143328 1159 raft_consensus.cc:491] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:34.143595 1159 raft_consensus.cc:513] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:34.144563 1159 leader_election.cc:290] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
W20250623 08:04:34.147830 32453 leader_election.cc:343] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:34.148178 32453 leader_election.cc:343] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:34.148913 32453 leader_election.cc:343] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.148890 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "205da880685b49b3b8669f73dc2c563b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.149493 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63029399994480ba7ccabcadc1ee5e0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:34.153153 32453 leader_election.cc:343] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:34.153362 32453 leader_election.cc:343] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:34.168115 32045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250623 08:04:34.169090 32045 consensus_peers.cc:487] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 -> Peer 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313): Couldn't send request to peer 9752155c20ee42fbb7ff7b7d5b682444. Status: Network error: Client connection negotiation failed: client connection to 127.31.69.61:33313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250623 08:04:34.181826 32453 leader_election.cc:343] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 5629d11c44e241d598f3e519b2fe1a44
I20250623 08:04:34.182636 1273 raft_consensus.cc:2747] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:34.183379 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9ea74d20c5f4eefbcb8508ed1774026" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.183818 32655 raft_consensus.cc:3058] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.187227 32655 raft_consensus.cc:2466] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:34.187455 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63029399994480ba7ccabcadc1ee5e0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.187637 32650 raft_consensus.cc:3058] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.193179 32650 raft_consensus.cc:2466] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:34.203588 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "205da880685b49b3b8669f73dc2c563b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.203799 32655 raft_consensus.cc:3058] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.214932 1273 consensus_queue.cc:1035] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:34.215574 1159 raft_consensus.cc:491] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.215838 1159 raft_consensus.cc:513] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.216688 1159 leader_election.cc:290] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.223094 32655 raft_consensus.cc:2466] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:34.225538 1275 leader_election.cc:290] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:34.227401 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dc696236a15420e9666cca708068fb2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:34.228001 32589 leader_election.cc:343] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.228245 32656 raft_consensus.cc:2466] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.234797 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0107b41d5c04a5c818f6792f2c18ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.238426 343 tablet_bootstrap.cc:654] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:34.244882 32314 consensus_peers.cc:487] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:34.215848 32453 leader_election.cc:304] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.262902 1159 raft_consensus.cc:2802] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.263382 1159 raft_consensus.cc:695] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:34.263746 1159 consensus_queue.cc:237] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:34.270203 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 54236.1us
I20250623 08:04:34.272282 32453 leader_election.cc:304] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.273892 1159 raft_consensus.cc:2802] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.275323 1159 raft_consensus.cc:695] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:34.275718 1159 consensus_queue.cc:237] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.279337 32376 raft_consensus.cc:3058] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.283995 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f97f14cbaffb46d681d8b11c0213cae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.285167 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 344ebe1a14bc4d3389b0161ff102f05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.291836 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0107b41d5c04a5c818f6792f2c18ce5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:34.292541 32453 leader_election.cc:343] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 5629d11c44e241d598f3e519b2fe1a44
I20250623 08:04:34.294735 32453 leader_election.cc:304] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.294282 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.295763 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3136ec837bea4619ae3995ceec9b24d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.296053 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea77b97018b74708bd6057bd2d65f90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.296315 32487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9891997e026448e39943ae13082bfe03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.297598 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac83131e6854890b803b3d89ded7cf8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:34.297972 1054 thread.cc:640] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.001s sys 0.000s
W20250623 08:04:34.298144 1054 thread.cc:606] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s user 0.001s sys 0.000s
I20250623 08:04:34.298400 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92005abc61dd40ef9da8124cadc29b6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.300284 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8111e6117c1541ab8905d5a5a7e37292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.300532 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11bb55088ab24b94af3e509810890247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.300751 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 36) took 3342 ms. Trace:
I20250623 08:04:34.300882 952 rpcz_store.cc:276] 0623 08:04:30.958167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.958805 (+ 638us) service_pool.cc:225] Handling call
0623 08:04:33.612222 (+2653417us) ts_tablet_manager.cc:1145] Deleting tablet 279624b8bb834b95a4f08045968ebfb4
0623 08:04:33.612232 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.684107 (+ 71875us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:33.753653 (+ 69546us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.300708 (+547055us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2652996}
I20250623 08:04:34.301386 952 tablet_replica.cc:331] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:34.301637 952 raft_consensus.cc:2241] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.301836 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.304805 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90ceb0f8d5c94c4e9427042d8a35e16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.305097 1159 raft_consensus.cc:2747] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:34.305663 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34da529ffde4595a23fc6596b30e3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.308243 952 raft_consensus.cc:2270] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.308880 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b04e38d3da44301a450fe1fb5e2ada6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.309489 952 ts_tablet_manager.cc:1905] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.312312 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b97f3019b893441bae18146fef094bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.315351 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89bb7ea31b4d4ce7a3c063853142fbb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.326467 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:34.327199 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:34.327636 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:34.328243 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_049497d4290c4bdfaf1e609b8b2ac9ac [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 2 got 0
I20250623 08:04:34.328446 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:34.328677 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:34.329171 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:34.329393 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:34.329756 32050 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_32db9f68dd974fb8aff372a8eeccce3d [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 1 got 0
I20250623 08:04:34.330078 979 ts_tablet_manager.cc:1918] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.330257 979 log.cc:1199] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/ae7c2ac546114a819a543f1667fb7844
I20250623 08:04:34.329761 32453 leader_election.cc:304] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.331895 343 tablet_bootstrap.cc:492] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.332239 1159 raft_consensus.cc:2802] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.332480 1159 raft_consensus.cc:695] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:34.333484 343 ts_tablet_manager.cc:1397] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.252s user 0.006s sys 0.003s
I20250623 08:04:34.333505 1159 consensus_queue.cc:237] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.334741 343 raft_consensus.cc:357] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.337102 343 raft_consensus.cc:383] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.337414 1275 raft_consensus.cc:491] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.337522 343 raft_consensus.cc:738] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.337584 1275 raft_consensus.cc:513] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.337893 32453 leader_election.cc:304] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.338233 1275 leader_election.cc:290] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:34.338208 343 consensus_queue.cc:260] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.338575 1159 raft_consensus.cc:2802] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.338979 1159 raft_consensus.cc:491] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.339164 1159 raft_consensus.cc:3058] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.339213 979 ts_tablet_manager.cc:1939] T ae7c2ac546114a819a543f1667fb7844 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:34.340595 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 72) took 3326 ms. Trace:
I20250623 08:04:34.340794 343 ts_tablet_manager.cc:1428] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:34.343039 343 tablet_bootstrap.cc:492] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:34.340765 979 rpcz_store.cc:276] 0623 08:04:31.014509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.014550 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:33.554259 (+2539709us) ts_tablet_manager.cc:1145] Deleting tablet ae7c2ac546114a819a543f1667fb7844
0623 08:04:33.554270 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:33.701456 (+147186us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.258628 (+557172us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.340544 (+ 81916us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2539547}
I20250623 08:04:34.344638 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet ae7c2ac546114a819a543f1667fb7844 (table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]) successfully deleted
I20250623 08:04:34.345124 343 tablet_bootstrap.cc:654] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.345229 979 tablet_replica.cc:331] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:34.345854 979 raft_consensus.cc:2241] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.346617 979 raft_consensus.cc:2270] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.347613 979 ts_tablet_manager.cc:1905] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.350867 1054 raft_consensus.cc:491] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.351047 1054 raft_consensus.cc:513] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.351660 1054 leader_election.cc:290] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.352751 343 tablet_bootstrap.cc:492] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.354208 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet a25bdc9b7e48467fba3771365e58bcbc
W20250623 08:04:34.359040 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 136e482995a24c5daec55a701f3091fa consensus state has no leader
I20250623 08:04:34.359237 32051 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 192) took 1706 ms. Trace:
I20250623 08:04:34.359328 32051 rpcz_store.cc:276] 0623 08:04:32.652745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.419903 (+767158us) service_pool.cc:225] Handling call
0623 08:04:33.420287 (+ 384us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:33.420511 (+ 224us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:33.420698 (+ 187us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:33.420769 (+ 71us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:33.449415 (+ 28646us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:34.359180 (+909765us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:33.429477 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:33.429600 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:33.429608 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:33.429612 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:33.429701 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:33.429713 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:33.429720 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:33.429779 (+ 59us) tablet.cc:699] Row locks acquired
0623 08:04:33.429783 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:33.429812 (+ 29us) write_op.cc:270] Start()
0623 08:04:33.429841 (+ 29us) write_op.cc:276] Timestamp: P: 1750665873429776 usec, L: 0
0623 08:04:33.429846 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:33.430001 (+ 155us) log.cc:844] Serialized 405 byte log entry
0623 08:04:33.441092 (+ 11091us) op_driver.cc:464] REPLICATION: finished
0623 08:04:33.442040 (+ 948us) write_op.cc:301] APPLY: starting
0623 08:04:33.442050 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:33.442078 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:33.442102 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:33.442141 (+ 39us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:33.442168 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:33.442175 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:33.442229 (+ 54us) log.cc:844] Serialized 23 byte log entry
0623 08:04:33.442273 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.442284 (+ 11us) write_op.cc:454] Released schema lock
0623 08:04:33.442290 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:33.442396 (+ 106us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.442400 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":918,"apply.run_cpu_time_us":336,"apply.run_wall_time_us":336,"num_ops":1,"prepare.queue_time_us":8212,"prepare.run_cpu_time_us":624,"prepare.run_wall_time_us":625,"raft.queue_time_us":2400,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":7,"replication_time_us":11228}]]}
I20250623 08:04:34.362879 32052 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e [id=7a569a9fe3bd4c608ddca48a734df1da] alter complete (version 2)
I20250623 08:04:34.364022 32056 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 92005abc61dd40ef9da8124cadc29b6a
I20250623 08:04:34.364279 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0
I20250623 08:04:34.364598 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet cdec6e9573b041e0a57d1e0e0239d97f
I20250623 08:04:34.354943 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet a25bdc9b7e48467fba3771365e58bcbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.367312 343 ts_tablet_manager.cc:1397] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20250623 08:04:34.368268 343 raft_consensus.cc:357] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.368482 343 raft_consensus.cc:383] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.368569 343 raft_consensus.cc:738] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.377336 979 ts_tablet_manager.cc:1918] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.377489 979 log.cc:1199] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/776b412b3f564a7581c6a5e1c84e8499
I20250623 08:04:34.377925 979 ts_tablet_manager.cc:1939] T 776b412b3f564a7581c6a5e1c84e8499 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:34.379530 343 consensus_queue.cc:260] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.381317 343 ts_tablet_manager.cc:1428] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250623 08:04:34.384022 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 73) took 3369 ms. Trace:
I20250623 08:04:34.384136 979 rpcz_store.cc:276] 0623 08:04:31.014877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.014914 (+ 37us) service_pool.cc:225] Handling call
0623 08:04:34.345158 (+3330244us) ts_tablet_manager.cc:1145] Deleting tablet 776b412b3f564a7581c6a5e1c84e8499
0623 08:04:34.345170 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.354415 (+ 9245us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.377207 (+ 22792us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.383963 (+ 6756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3330081}
I20250623 08:04:34.385476 979 tablet_replica.cc:331] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:34.385663 979 raft_consensus.cc:2241] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.385839 979 raft_consensus.cc:2270] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.387655 343 tablet_bootstrap.cc:492] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:34.389981 979 ts_tablet_manager.cc:1905] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.393656 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 776b412b3f564a7581c6a5e1c84e8499 (table default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652]) successfully deleted
I20250623 08:04:34.394447 32052 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_75451c4863a34537a328617c53d46cae [id=be88c6ecf20d4ab1b3757c27ce998f87] alter complete (version 2)
I20250623 08:04:34.396386 343 tablet_bootstrap.cc:654] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.403906 1054 raft_consensus.cc:491] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.405838 32055 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_977fe8fc665f4267b394065e22f67eea [id=be9754ecb40a413d9940224adbd7e2eb] alter complete (version 2)
I20250623 08:04:34.410652 32050 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_d496fd69b6f140b88b5abc86c8839ee2 [id=6c57db68f5ed45a3b64d1e3a02ce8a2a] alter complete (version 1)
I20250623 08:04:34.413806 32059 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 199) took 1492 ms. Trace:
I20250623 08:04:34.413940 32059 rpcz_store.cc:276] 0623 08:04:32.921567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.512096 (+590529us) service_pool.cc:225] Handling call
0623 08:04:33.524324 (+ 12228us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fa306 00007f19c7644956 00007f19c7644685 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:33.524926 (+ 602us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:33.525098 (+ 172us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:33.525166 (+ 68us) catalog_manager.cc:2799] Locking tablets
0623 08:04:33.742993 (+217827us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:33.770774 (+ 27781us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:33.770859 (+ 85us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:34.364241 (+593382us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:34.413730 (+ 49489us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:33.743681 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:33.743796 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:33.743803 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:33.743805 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:33.743919 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:33.743933 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:33.743937 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:33.744040 (+ 103us) tablet.cc:699] Row locks acquired
0623 08:04:33.744042 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:33.744078 (+ 36us) write_op.cc:270] Start()
0623 08:04:33.744109 (+ 31us) write_op.cc:276] Timestamp: P: 1750665873744004 usec, L: 0
0623 08:04:33.744113 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:33.744300 (+ 187us) log.cc:844] Serialized 725 byte log entry
0623 08:04:33.768873 (+ 24573us) op_driver.cc:464] REPLICATION: finished
0623 08:04:33.769199 (+ 326us) write_op.cc:301] APPLY: starting
0623 08:04:33.769221 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:33.769281 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:33.769351 (+ 70us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:33.769510 (+ 159us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:33.769563 (+ 53us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:33.769575 (+ 12us) write_op.cc:312] APPLY: finished
0623 08:04:33.769743 (+ 168us) log.cc:844] Serialized 29 byte log entry
0623 08:04:33.769835 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.769861 (+ 26us) write_op.cc:454] Released schema lock
0623 08:04:33.769871 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:33.769963 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.769969 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":27374976,"child_traces":[["op",{"apply.queue_time_us":247,"apply.run_cpu_time_us":750,"apply.run_wall_time_us":750,"num_ops":2,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":719,"prepare.run_wall_time_us":719,"raft.queue_time_us":87,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":24714}]]}
I20250623 08:04:34.411756 1054 raft_consensus.cc:513] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.415433 32056 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac" } new_table_name: "default.Table_e192d00b3ef4472c8dbe2a9fdadc0290" modify_external_catalogs: true
W20250623 08:04:34.416949 32054 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet e602d391526d4b0eb0dbf5189afbe075 consensus state has no leader
I20250623 08:04:34.417270 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dc696236a15420e9666cca708068fb2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:34.417421 32053 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 178) took 2166 ms. Trace:
I20250623 08:04:34.417582 1159 raft_consensus.cc:513] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:34.418078 32054 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 736eb56dae194df2af90bf33d3294a4e consensus state has no leader
W20250623 08:04:34.418419 32054 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 5f1fb8231e8e45208ac4f412134d8696 consensus state has no leader
I20250623 08:04:34.418867 1054 leader_election.cc:290] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.417543 32053 rpcz_store.cc:276] 0623 08:04:32.251305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.481216 (+229911us) service_pool.cc:225] Handling call
0623 08:04:32.481674 (+ 458us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:32.481856 (+ 182us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:32.481932 (+ 76us) catalog_manager.cc:2799] Locking tablets
0623 08:04:32.707003 (+225071us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:32.758573 (+ 51570us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:32.758696 (+ 123us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:34.364534 (+1605838us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:34.417369 (+ 52835us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:32.708825 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:32.708936 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:32.708943 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:32.708945 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:32.709074 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:32.709085 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:32.709089 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:32.709193 (+ 104us) tablet.cc:699] Row locks acquired
0623 08:04:32.709196 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:32.709234 (+ 38us) write_op.cc:270] Start()
0623 08:04:32.709271 (+ 37us) write_op.cc:276] Timestamp: P: 1750665872709163 usec, L: 0
0623 08:04:32.709276 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:32.709496 (+ 220us) log.cc:844] Serialized 725 byte log entry
0623 08:04:32.757391 (+ 47895us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.757646 (+ 255us) write_op.cc:301] APPLY: starting
0623 08:04:32.757665 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:32.757713 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:32.757789 (+ 76us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:32.757930 (+ 141us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:32.757976 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:32.757988 (+ 12us) write_op.cc:312] APPLY: finished
0623 08:04:32.758085 (+ 97us) log.cc:844] Serialized 29 byte log entry
0623 08:04:32.758448 (+ 363us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.758469 (+ 21us) write_op.cc:454] Released schema lock
0623 08:04:32.758478 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":165,"apply.run_cpu_time_us":620,"apply.run_wall_time_us":1111,"mutex_wait_us":6610,"num_ops":2,"prepare.queue_time_us":992,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":20453,"raft.queue_time_us":6713,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":48088}]]}
I20250623 08:04:34.419467 32513 raft_consensus.cc:2391] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
W20250623 08:04:34.419526 32054 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 18ea63263b9b466e922169eecafab747 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 18ea63263b9b466e922169eecafab747 consensus state has no leader
I20250623 08:04:34.419842 32053 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_cc3ded2eef374e8d9f476cc87bfc2470" } new_table_name: "default.Table_aaa3ca8d9bb0406981f0dfff8cbf9e3c" modify_external_catalogs: true
I20250623 08:04:34.420184 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a0b9402a264800a1ea39707c10da0c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:34.420917 32054 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.1:45613 (request call id 153) took 1816 ms (client timeout 1000 ms). Trace:
I20250623 08:04:34.422055 342 tablet_bootstrap.cc:492] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:34.422294 1054 raft_consensus.cc:491] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.424501 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.425020 979 ts_tablet_manager.cc:1918] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.421156 32059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c2afde03c007450390a0466fe1d39344" } new_table_name: "default.Table_912f5673d23f4f0797a33298858f2789" modify_external_catalogs: true
I20250623 08:04:34.425319 342 ts_tablet_manager.cc:1397] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.383s user 0.021s sys 0.128s
I20250623 08:04:34.425324 1054 raft_consensus.cc:513] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.426050 1054 leader_election.cc:290] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.426134 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 92005abc61dd40ef9da8124cadc29b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.425246 1159 leader_election.cc:290] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.426376 342 raft_consensus.cc:357] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.426581 342 raft_consensus.cc:383] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.426676 342 raft_consensus.cc:738] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.426971 342 consensus_queue.cc:260] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.427526 342 ts_tablet_manager.cc:1428] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:34.427887 342 tablet_bootstrap.cc:492] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:34.429555 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712abfca1fa4489f882534c33b9db1be" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.429701 342 tablet_bootstrap.cc:654] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.429759 32656 raft_consensus.cc:2466] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.431180 979 log.cc:1199] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/9918b15e48144ef0b8f3aedc2135be59
I20250623 08:04:34.431773 979 ts_tablet_manager.cc:1939] T 9918b15e48144ef0b8f3aedc2135be59 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:34.432235 952 ts_tablet_manager.cc:1918] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.433490 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9c8d480b8ba49a1be140b0e0a79f4e4" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.433673 32656 raft_consensus.cc:2466] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:34.433734 32054 rpcz_store.cc:269] 0623 08:04:32.604357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.701336 (+ 96979us) service_pool.cc:225] Handling call
0623 08:04:34.420863 (+1719527us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:33.409201 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:33.409327 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:33.409333 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:33.409336 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:33.409480 (+ 144us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:33.409495 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:33.409500 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:33.409623 (+ 123us) tablet.cc:699] Row locks acquired
0623 08:04:33.409626 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:33.409664 (+ 38us) write_op.cc:270] Start()
0623 08:04:33.409701 (+ 37us) write_op.cc:276] Timestamp: P: 1750665873409628 usec, L: 0
0623 08:04:33.409705 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:33.409913 (+ 208us) log.cc:844] Serialized 901 byte log entry
0623 08:04:33.414622 (+ 4709us) op_driver.cc:464] REPLICATION: finished
0623 08:04:33.414820 (+ 198us) write_op.cc:301] APPLY: starting
0623 08:04:33.414838 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:33.414891 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:33.414933 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:33.415023 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:33.415074 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:33.415085 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:33.415190 (+ 105us) log.cc:844] Serialized 29 byte log entry
0623 08:04:33.415264 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.415310 (+ 46us) write_op.cc:454] Released schema lock
0623 08:04:33.415319 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:33.415423 (+ 104us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.415428 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":135,"child_traces":[["op",{"apply.queue_time_us":141,"apply.run_cpu_time_us":574,"apply.run_wall_time_us":573,"num_ops":2,"prepare.queue_time_us":4233,"prepare.run_cpu_time_us":853,"prepare.run_wall_time_us":853,"raft.queue_time_us":905,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":299,"replication_time_us":4881,"spinlock_wait_cycles":80768}]]}
W20250623 08:04:34.435465 32055 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:39129 (request call id 169) took 2447 ms (client timeout 1000 ms). Trace:
W20250623 08:04:34.435853 32055 rpcz_store.cc:269] 0623 08:04:31.987798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.990061 (+ 2263us) service_pool.cc:225] Handling call
0623 08:04:34.435425 (+2445364us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:32.479398 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:32.634114 (+154716us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:32.634121 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:32.634124 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:32.634325 (+ 201us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:32.634341 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:32.634346 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:32.634421 (+ 75us) tablet.cc:699] Row locks acquired
0623 08:04:32.634425 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:32.634472 (+ 47us) write_op.cc:270] Start()
0623 08:04:32.634508 (+ 36us) write_op.cc:276] Timestamp: P: 1750665872634408 usec, L: 0
0623 08:04:32.634512 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:32.634724 (+ 212us) log.cc:844] Serialized 527 byte log entry
0623 08:04:32.697096 (+ 62372us) op_driver.cc:464] REPLICATION: finished
0623 08:04:32.697252 (+ 156us) write_op.cc:301] APPLY: starting
0623 08:04:32.697272 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:32.697316 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:32.697345 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:32.697419 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:32.697455 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:32.697462 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:32.697554 (+ 92us) log.cc:844] Serialized 23 byte log entry
0623 08:04:32.697632 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.697653 (+ 21us) write_op.cc:454] Released schema lock
0623 08:04:32.697660 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:32.697734 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:32.697738 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:34.380782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.380904 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.380910 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:34.380912 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:34.381002 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.381012 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.381016 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:34.381071 (+ 55us) tablet.cc:699] Row locks acquired
0623 08:04:34.381073 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:34.381103 (+ 30us) write_op.cc:270] Start()
0623 08:04:34.381129 (+ 26us) write_op.cc:276] Timestamp: P: 1750665874381074 usec, L: 0
0623 08:04:34.381132 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.381261 (+ 129us) log.cc:844] Serialized 356 byte log entry
0623 08:04:34.403950 (+ 22689us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.404100 (+ 150us) write_op.cc:301] APPLY: starting
0623 08:04:34.404118 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.404159 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.404191 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.404245 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.404281 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:34.404288 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:34.404362 (+ 74us) log.cc:844] Serialized 23 byte log entry
0623 08:04:34.404426 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.404442 (+ 16us) write_op.cc:454] Released schema lock
0623 08:04:34.404448 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:34.404514 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.404518 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":92,"apply.run_cpu_time_us":477,"apply.run_wall_time_us":476,"num_ops":1,"prepare.queue_time_us":206976,"prepare.run_cpu_time_us":786,"prepare.run_wall_time_us":155800,"raft.queue_time_us":519,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":299,"replication_time_us":62547}],["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":409,"apply.run_wall_time_us":408,"num_ops":1,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":577,"prepare.run_wall_time_us":576,"raft.queue_time_us":511,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":22785}]]}
I20250623 08:04:34.437155 1278 raft_consensus.cc:491] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.437300 1278 raft_consensus.cc:513] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.437448 32055 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6ca32a3252134e6587bca37cff71c969" } new_table_name: "default.Table_811a9724f3234933a3f6352f67b17e73" modify_external_catalogs: true
I20250623 08:04:34.438045 1278 leader_election.cc:290] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:34.439457 343 tablet_bootstrap.cc:492] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.440454 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_2092a828aea549a38d1bd42248c8804c" } modify_external_catalogs: true
I20250623 08:04:34.442847 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 74) took 3427 ms. Trace:
I20250623 08:04:34.443409 979 rpcz_store.cc:276] 0623 08:04:31.015192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.015231 (+ 39us) service_pool.cc:225] Handling call
0623 08:04:34.385396 (+3370165us) ts_tablet_manager.cc:1145] Deleting tablet 9918b15e48144ef0b8f3aedc2135be59
0623 08:04:34.385405 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.393662 (+ 8257us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.424847 (+ 31185us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.442812 (+ 17965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3370022}
I20250623 08:04:34.444350 979 tablet_replica.cc:331] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:34.444933 979 raft_consensus.cc:2241] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.445499 979 raft_consensus.cc:2270] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.446347 979 ts_tablet_manager.cc:1905] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.448500 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dc696236a15420e9666cca708068fb2" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.448761 32656 raft_consensus.cc:3058] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.451684 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet cdec6e9573b041e0a57d1e0e0239d97f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.451908 32656 raft_consensus.cc:2466] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:34.451982 343 ts_tablet_manager.cc:1397] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.005s
W20250623 08:04:34.452200 32044 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 736eb56dae194df2af90bf33d3294a4e consensus state has no leader
I20250623 08:04:34.452836 32589 leader_election.cc:304] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.442147 342 tablet_bootstrap.cc:492] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:34.453577 342 ts_tablet_manager.cc:1397] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20250623 08:04:34.453996 952 log.cc:1199] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/32c82141edea4068b54755a85c6d7fb8
I20250623 08:04:34.454789 342 raft_consensus.cc:357] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.455152 342 raft_consensus.cc:383] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.455255 342 raft_consensus.cc:738] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.455482 342 consensus_queue.cc:260] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.456044 342 ts_tablet_manager.cc:1428] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:34.456349 342 tablet_bootstrap.cc:492] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:34.457057 952 ts_tablet_manager.cc:1939] T 32c82141edea4068b54755a85c6d7fb8 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:34.457537 32498 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:34.458698 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 40) took 3499 ms. Trace:
I20250623 08:04:34.459120 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_2092a828aea549a38d1bd42248c8804c" } modify_external_catalogs: true
I20250623 08:04:34.453326 1278 raft_consensus.cc:2747] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:34.454849 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 966a159e212040109e2842c15b69fb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.453213 343 raft_consensus.cc:357] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.460451 343 raft_consensus.cc:383] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.460543 343 raft_consensus.cc:738] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.459008 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet cdec6e9573b041e0a57d1e0e0239d97f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:34.458158 342 tablet_bootstrap.cc:654] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.460942 952 rpcz_store.cc:276] 0623 08:04:30.959666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.960530 (+ 864us) service_pool.cc:225] Handling call
0623 08:04:34.301300 (+3340770us) ts_tablet_manager.cc:1145] Deleting tablet 32c82141edea4068b54755a85c6d7fb8
0623 08:04:34.301310 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.357641 (+ 56331us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.432068 (+ 74427us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.458636 (+ 26568us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3649,"tablet-delete.queue_time_us":3340542}
I20250623 08:04:34.460024 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 92005abc61dd40ef9da8124cadc29b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:34.459952 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet a25bdc9b7e48467fba3771365e58bcbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:34.462868 343 consensus_queue.cc:260] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.463073 952 tablet_replica.cc:331] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:34.463805 343 ts_tablet_manager.cc:1428] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250623 08:04:34.464565 952 raft_consensus.cc:2241] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.465310 952 raft_consensus.cc:2270] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:34.461906 32453 leader_election.cc:343] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 79731106915d4cebb9dcc3c10386ffa9
I20250623 08:04:34.463999 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f9e0d1b85a94ae9ae86e53688a55db5) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.464532 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.465013 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet f0117cbb09dd4067836266c9d6a32adf with delete_type TABLET_DATA_DELETED (replaced by tablet 2ecf4af9e476495683bd63cac20d4672) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.465487 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.465940 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet c6a97b34e386443fa19a9601621b5017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.470682 343 tablet_bootstrap.cc:492] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:34.466706 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 9918b15e48144ef0b8f3aedc2135be59 (table default.Table_c09ee28ff44c4c43835c2faa880a1c86 [id=371e56649fc6441c8aa52739de5aa077]) successfully deleted
I20250623 08:04:34.473702 979 ts_tablet_manager.cc:1918] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.475412 979 log.cc:1199] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/cde7df7d39a54114a4dcb28cfd1eb429
I20250623 08:04:34.475941 979 ts_tablet_manager.cc:1939] T cde7df7d39a54114a4dcb28cfd1eb429 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:34.476850 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 75) took 3461 ms. Trace:
I20250623 08:04:34.477241 979 rpcz_store.cc:276] 0623 08:04:31.015495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.015532 (+ 37us) service_pool.cc:225] Handling call
0623 08:04:34.444264 (+3428732us) ts_tablet_manager.cc:1145] Deleting tablet cde7df7d39a54114a4dcb28cfd1eb429
0623 08:04:34.444276 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.450677 (+ 6401us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.469638 (+ 18961us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.476782 (+ 7144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3428571}
I20250623 08:04:34.478226 979 tablet_replica.cc:331] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:34.478699 979 raft_consensus.cc:2241] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.479588 979 raft_consensus.cc:2270] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.480306 979 ts_tablet_manager.cc:1905] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.481848 343 tablet_bootstrap.cc:654] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.485816 1303 raft_consensus.cc:491] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.487924 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79731106915d4cebb9dcc3c10386ffa9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.491573 343 tablet_bootstrap.cc:492] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.491545 1303 raft_consensus.cc:513] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.491750 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet cde7df7d39a54114a4dcb28cfd1eb429 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]) successfully deleted
I20250623 08:04:34.492664 343 ts_tablet_manager.cc:1397] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250623 08:04:34.493743 343 raft_consensus.cc:357] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.495864 343 raft_consensus.cc:383] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.496277 343 raft_consensus.cc:738] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.496758 343 consensus_queue.cc:260] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.497895 343 ts_tablet_manager.cc:1428] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20250623 08:04:34.502918 32055 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9891997e026448e39943ae13082bfe03 consensus state has no leader
I20250623 08:04:34.503800 32055 master_service.cc:948] ReplaceTablet: received request to replace tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:34.503895 343 tablet_bootstrap.cc:492] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:34.505198 952 ts_tablet_manager.cc:1905] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.505159 848 consensus_queue.cc:1035] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250623 08:04:34.505182 1159 consensus_queue.cc:1035] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20250623 08:04:34.507154 32453 leader_election.cc:304] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.507898 32453 leader_election.cc:304] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.508189 1159 raft_consensus.cc:2802] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.508137 848 consensus_queue.cc:1035] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250623 08:04:34.508301 1159 raft_consensus.cc:491] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.508507 342 tablet_bootstrap.cc:492] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:34.509047 343 tablet_bootstrap.cc:654] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.509945 1159 raft_consensus.cc:3058] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.510253 657 raft_consensus.cc:2802] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.510339 657 raft_consensus.cc:491] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.510659 657 raft_consensus.cc:3058] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.511354 342 ts_tablet_manager.cc:1397] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.003s
I20250623 08:04:34.512152 32453 leader_election.cc:304] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.512454 848 raft_consensus.cc:2802] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.512663 848 raft_consensus.cc:695] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:34.512966 848 consensus_queue.cc:237] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.515329 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f62ec83c88e43438b963e740d2c7004" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:34.517096 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 32c82141edea4068b54755a85c6d7fb8 (table default.Table_c30b48097bd643b18b2a744839780b9e [id=7e59ad2235c1418d97f2b73ff2a48300]) successfully deleted
I20250623 08:04:34.520813 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a0b9402a264800a1ea39707c10da0c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.521445 1303 leader_election.cc:290] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:34.521728 32377 thread.cc:640] prepare [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.002s sys 0.038s
I20250623 08:04:34.521739 342 raft_consensus.cc:357] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.522145 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f62ec83c88e43438b963e740d2c7004" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.522060 342 raft_consensus.cc:383] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.522569 342 raft_consensus.cc:738] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.525928 342 consensus_queue.cc:260] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.526197 32515 raft_consensus.cc:2466] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:34.526276 981 ts_tablet_manager.cc:1918] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.526681 981 log.cc:1199] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/32c82141edea4068b54755a85c6d7fb8
I20250623 08:04:34.527129 981 ts_tablet_manager.cc:1939] T 32c82141edea4068b54755a85c6d7fb8 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:34.527598 342 ts_tablet_manager.cc:1428] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250623 08:04:34.528069 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 79) took 2114 ms. Trace:
I20250623 08:04:34.528647 342 tablet_bootstrap.cc:492] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:34.530997 981 rpcz_store.cc:276] 0623 08:04:32.413761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.413811 (+ 50us) service_pool.cc:225] Handling call
0623 08:04:33.782901 (+1369090us) ts_tablet_manager.cc:1145] Deleting tablet 32c82141edea4068b54755a85c6d7fb8
0623 08:04:33.782912 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.256353 (+473441us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.526133 (+269780us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.527891 (+ 1758us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14034,"tablet-delete.queue_time_us":1368802}
W20250623 08:04:34.529770 339 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.016s sys 0.015s
W20250623 08:04:34.531693 339 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.538s user 0.016s sys 0.015s
I20250623 08:04:34.532473 339 tablet_bootstrap.cc:492] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:34.532622 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 78) took 2119 ms. Trace:
I20250623 08:04:34.532936 339 ts_tablet_manager.cc:1397] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.542s user 0.016s sys 0.015s
I20250623 08:04:34.533025 981 rpcz_store.cc:276] 0623 08:04:32.413648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.414133 (+ 485us) service_pool.cc:225] Handling call
0623 08:04:34.531424 (+2117291us) ts_tablet_manager.cc:1145] Deleting tablet 79731106915d4cebb9dcc3c10386ffa9
0623 08:04:34.531434 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.532574 (+ 1140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2117051}
I20250623 08:04:34.536913 981 tablet_replica.cc:331] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:34.537032 339 raft_consensus.cc:357] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:34.538663 32589 leader_election.cc:343] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.537164 981 raft_consensus.cc:2241] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:34.539774 339 raft_consensus.cc:383] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.539892 339 raft_consensus.cc:738] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.541092 339 consensus_queue.cc:260] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.541985 339 ts_tablet_manager.cc:1428] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:34.543846 342 tablet_bootstrap.cc:654] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.542294 339 tablet_bootstrap.cc:492] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:34.546617 339 tablet_bootstrap.cc:654] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.548255 1159 raft_consensus.cc:513] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.549261 892 rpc_server.cc:307] RPC server started. Bound to: 127.31.69.61:33313
I20250623 08:04:34.551483 657 raft_consensus.cc:513] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.552556 657 leader_election.cc:290] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.554231 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:34.554797 32020 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskJxinPB/build/asan/bin/kudu as pid 892
I20250623 08:04:34.555102 1159 leader_election.cc:290] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.557897 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
W20250623 08:04:34.559777 32589 leader_election.cc:343] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.559932 32589 leader_election.cc:304] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.560240 1278 raft_consensus.cc:2747] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:34.560472 32589 leader_election.cc:343] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.560492 952 ts_tablet_manager.cc:1918] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.560673 981 raft_consensus.cc:2270] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:34.560791 32589 consensus_peers.cc:487] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:34.560631 952 log.cc:1199] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/726d80e05a344f27b5a289723cb5aed7
I20250623 08:04:34.561136 32589 leader_election.cc:304] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.561398 1278 raft_consensus.cc:2802] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.561525 1278 raft_consensus.cc:491] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.561645 1278 raft_consensus.cc:3058] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.562151 952 ts_tablet_manager.cc:1939] T 726d80e05a344f27b5a289723cb5aed7 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:34.563848 979 ts_tablet_manager.cc:1918] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.564091 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7adec1482119405daca4aa603770ae22" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.564200 979 log.cc:1199] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/4ea81642050141a9be84f151a5785abc
I20250623 08:04:34.564323 32654 raft_consensus.cc:2466] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.566264 979 ts_tablet_manager.cc:1939] T 4ea81642050141a9be84f151a5785abc P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:34.568017 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9c8d480b8ba49a1be140b0e0a79f4e4" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.568204 32649 raft_consensus.cc:3058] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.568284 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712abfca1fa4489f882534c33b9db1be" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.568462 32653 raft_consensus.cc:3058] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.569748 32453 leader_election.cc:304] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:34.570503 32453 consensus_peers.cc:487] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:34.570832 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 76) took 3551 ms. Trace:
I20250623 08:04:34.570977 979 rpcz_store.cc:276] 0623 08:04:31.015800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.015835 (+ 35us) service_pool.cc:225] Handling call
0623 08:04:34.478123 (+3462288us) ts_tablet_manager.cc:1145] Deleting tablet 4ea81642050141a9be84f151a5785abc
0623 08:04:34.478133 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.502455 (+ 24322us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.559601 (+ 57146us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.567144 (+ 7543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3462142}
I20250623 08:04:34.571712 979 tablet_replica.cc:331] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:34.572005 979 raft_consensus.cc:2241] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:34.572343 979 pending_rounds.cc:70] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:34.572477 979 pending_rounds.cc:77] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727395871035392 op_type: NO_OP noop_request { }
I20250623 08:04:34.572316 32653 raft_consensus.cc:2466] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:34.572649 979 raft_consensus.cc:2887] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:34.572860 979 raft_consensus.cc:2270] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:34.573134 32453 consensus_peers.cc:487] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cde7df7d39a54114a4dcb28cfd1eb429. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:34.573204 32649 raft_consensus.cc:2466] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:34.573333 1278 raft_consensus.cc:513] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.574295 1278 leader_election.cc:290] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:34.570689 1159 raft_consensus.cc:2802] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.574615 1159 raft_consensus.cc:491] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.574698 32453 leader_election.cc:304] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.574769 1159 raft_consensus.cc:3058] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.575006 32453 leader_election.cc:304] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.575137 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f62ec83c88e43438b963e740d2c7004" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.575546 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f62ec83c88e43438b963e740d2c7004" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
W20250623 08:04:34.575728 32589 leader_election.cc:343] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.575757 32515 raft_consensus.cc:3058] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.575778 979 ts_tablet_manager.cc:1905] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.577817 1159 raft_consensus.cc:513] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.578258 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:34.578411 1159 leader_election.cc:290] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.578585 1159 raft_consensus.cc:2802] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.578594 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 136e482995a24c5daec55a701f3091fa (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:34.578913 1251 raft_consensus.cc:2802] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.579092 1251 raft_consensus.cc:695] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:34.579154 32515 raft_consensus.cc:2466] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:34.579440 1251 consensus_queue.cc:237] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.579839 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:34.580170 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:34.580408 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:34.582077 981 ts_tablet_manager.cc:1905] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.582839 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7adec1482119405daca4aa603770ae22" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.583123 32649 raft_consensus.cc:3058] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:34.537302 32377 thread.cc:606] prepare [worker] (thread pool) Time spent starting thread: real 0.514s user 0.002s sys 0.038s
I20250623 08:04:34.591081 32589 leader_election.cc:304] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.591113 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 41) took 3606 ms. Trace:
I20250623 08:04:34.591243 952 rpcz_store.cc:276] 0623 08:04:30.960900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.960945 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:34.462990 (+3502045us) ts_tablet_manager.cc:1145] Deleting tablet 726d80e05a344f27b5a289723cb5aed7
0623 08:04:34.463002 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.554023 (+ 91021us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.560048 (+ 6025us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.567249 (+ 7201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3501828}
I20250623 08:04:34.591585 952 tablet_replica.cc:331] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:34.591588 1278 raft_consensus.cc:2802] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.591838 1278 raft_consensus.cc:695] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:34.592103 1278 consensus_queue.cc:237] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.593590 952 raft_consensus.cc:2241] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.594895 32649 raft_consensus.cc:2466] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:34.595484 952 raft_consensus.cc:2270] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.596611 1159 raft_consensus.cc:695] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:34.597025 1159 consensus_queue.cc:237] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:34.593698 32453 consensus_peers.cc:487] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 776b412b3f564a7581c6a5e1c84e8499. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:34.598244 32453 leader_election.cc:304] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.598605 848 raft_consensus.cc:2802] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.598734 848 raft_consensus.cc:695] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:34.599061 848 consensus_queue.cc:237] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.596592 952 ts_tablet_manager.cc:1905] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.608809 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:34.609339 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:34.609874 32051 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:34.618660 32634 tablet_service.cc:1468] Processing CreateTablet for tablet de743139c4324faba19ff3e5cfac855d (DEFAULT_TABLE table=default.Table_d3115e0a02304f6d80807841b76504e4 [id=fc36d4cdd9684c6f84517d848ae3395b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.619127 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de743139c4324faba19ff3e5cfac855d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.619271 32632 tablet_service.cc:1468] Processing CreateTablet for tablet b61b02cf2904404f8c0ffef7d268fefc (DEFAULT_TABLE table=default.Table_3d6c2aca7a944301bc67f7901cfef8b3 [id=e1d4219a19fc4fffa81e7693469aebce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.619746 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61b02cf2904404f8c0ffef7d268fefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.551512 892 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-1/data/info.pb
I20250623 08:04:34.549759 1341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.69.61:33313 every 8 connection(s)
I20250623 08:04:34.625214 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 7dc5e0fbeba4412aa42d116cd8882238 (DEFAULT_TABLE table=default.Table_ea4d110c161f4b008acde5f22ffa3562 [id=79d117a74f18466ab87a1701e641e7da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.625608 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc5e0fbeba4412aa42d116cd8882238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.631822 979 ts_tablet_manager.cc:1918] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.632377 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 5f957f5575e9475eb9f639e3ad5cb809 (DEFAULT_TABLE table=default.Table_ba8a245f5a564519b522c175b4117f52 [id=4547bcef8b5d4d25acf8de6cbba4f7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.632958 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f957f5575e9475eb9f639e3ad5cb809. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.635455 32632 tablet_service.cc:1468] Processing CreateTablet for tablet a0b1287cf3b6459eafd032afc77c5824 (DEFAULT_TABLE table=default.Table_cc3b284f368e4123bf1a2da98f4f4fa8 [id=906debc01a6f45a49a86b5f939ac1dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.635959 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0b1287cf3b6459eafd032afc77c5824. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.645422 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet a25bdc9b7e48467fba3771365e58bcbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.646266 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.651990 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 59015204545e436cb558fe4465a67c97 (DEFAULT_TABLE table=default.Table_e541bb700f4c4e2abc268babf435c6b0 [id=9979c6d8878443c09e6ff9f4aea65d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.652558 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59015204545e436cb558fe4465a67c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.652873 979 log.cc:1199] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/696f5d2797ef463bb72a2119501c6ef3
I20250623 08:04:34.653488 979 ts_tablet_manager.cc:1939] T 696f5d2797ef463bb72a2119501c6ef3 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:34.654495 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 77) took 3638 ms. Trace:
I20250623 08:04:34.654665 979 rpcz_store.cc:276] 0623 08:04:31.016120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.016157 (+ 37us) service_pool.cc:225] Handling call
0623 08:04:34.571642 (+3555485us) ts_tablet_manager.cc:1145] Deleting tablet 696f5d2797ef463bb72a2119501c6ef3
0623 08:04:34.571650 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.584757 (+ 13107us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.631667 (+ 46910us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.654438 (+ 22771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3555337}
I20250623 08:04:34.655310 979 tablet_replica.cc:331] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:34.655651 979 raft_consensus.cc:2241] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.655898 979 raft_consensus.cc:2270] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.656549 979 ts_tablet_manager.cc:1905] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.660105 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet f3d32798168345edb9898751d083057c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.660640 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.661020 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ecf4af9e476495683bd63cac20d4672 with delete_type TABLET_DATA_DELETED (replaced by tablet 52d0a356c74f4daea9093c998b4bf73e) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.661407 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 38ecd4c243b840f4b66d8e01078dca64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.661816 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 2060a643d0da47efa868f52bdb331b1f with delete_type TABLET_DATA_DELETED (replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.662245 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.663322 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 6857437f85ab40f6a24e74ce3b5efbdc (DEFAULT_TABLE table=default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.663923 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6857437f85ab40f6a24e74ce3b5efbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.666221 343 tablet_bootstrap.cc:492] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.668031 343 ts_tablet_manager.cc:1397] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.164s user 0.006s sys 0.003s
I20250623 08:04:34.671751 343 raft_consensus.cc:357] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.672093 343 raft_consensus.cc:383] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.672245 343 raft_consensus.cc:738] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.672205 32630 tablet_service.cc:1468] Processing CreateTablet for tablet c74e8c697d244ac2ac8174bb1fc76e61 (DEFAULT_TABLE table=default.Table_5785d39307cb42758ded4062307c55e3 [id=4ceb1e1117f747b3a5424c01cd47bd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.672734 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c74e8c697d244ac2ac8174bb1fc76e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.680922 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 8f83a6d453ec4ffe85282b4385df5f11 (DEFAULT_TABLE table=default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.681455 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f83a6d453ec4ffe85282b4385df5f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.686172 32499 tablet_service.cc:1468] Processing CreateTablet for tablet de743139c4324faba19ff3e5cfac855d (DEFAULT_TABLE table=default.Table_d3115e0a02304f6d80807841b76504e4 [id=fc36d4cdd9684c6f84517d848ae3395b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.686640 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de743139c4324faba19ff3e5cfac855d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.672601 343 consensus_queue.cc:260] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.687888 343 ts_tablet_manager.cc:1428] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250623 08:04:34.688597 343 tablet_bootstrap.cc:492] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:34.689877 32631 tablet_service.cc:1468] Processing CreateTablet for tablet ea588a48e6164d3db148fdaa306268f6 (DEFAULT_TABLE table=default.Table_b523bec7182443139e891357d295e3a1 [id=1c4298a5f9014d8eb4ccad8c8fbdeeac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.690258 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea588a48e6164d3db148fdaa306268f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.691406 32494 tablet_service.cc:1468] Processing CreateTablet for tablet b61b02cf2904404f8c0ffef7d268fefc (DEFAULT_TABLE table=default.Table_3d6c2aca7a944301bc67f7901cfef8b3 [id=e1d4219a19fc4fffa81e7693469aebce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.691763 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61b02cf2904404f8c0ffef7d268fefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.693090 343 tablet_bootstrap.cc:654] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.698143 32630 tablet_service.cc:1468] Processing CreateTablet for tablet d5cd316e6f4447be958611ab07ed31fc (DEFAULT_TABLE table=default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac [id=793edd0cb585499197d2f0e57a83be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.698621 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5cd316e6f4447be958611ab07ed31fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.698621 32486 tablet_service.cc:1468] Processing CreateTablet for tablet a0b1287cf3b6459eafd032afc77c5824 (DEFAULT_TABLE table=default.Table_cc3b284f368e4123bf1a2da98f4f4fa8 [id=906debc01a6f45a49a86b5f939ac1dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.699036 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0b1287cf3b6459eafd032afc77c5824. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.700982 1353 sys_catalog.cc:263] Verifying existing consensus state
I20250623 08:04:34.703382 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 5f957f5575e9475eb9f639e3ad5cb809 (DEFAULT_TABLE table=default.Table_ba8a245f5a564519b522c175b4117f52 [id=4547bcef8b5d4d25acf8de6cbba4f7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.703487 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 23928d64bcef4c06af81698e688b7f94 (DEFAULT_TABLE table=default.Table_5c6bbeaad8274e05904301c82a72aa74 [id=ea13915f7ffd4e9c9116fb0b12fe0351]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.703779 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f957f5575e9475eb9f639e3ad5cb809. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.704082 343 tablet_bootstrap.cc:492] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.704246 343 ts_tablet_manager.cc:1397] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250623 08:04:34.704895 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23928d64bcef4c06af81698e688b7f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.705020 343 raft_consensus.cc:357] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.712452 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 6857437f85ab40f6a24e74ce3b5efbdc (DEFAULT_TABLE table=default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.712841 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6857437f85ab40f6a24e74ce3b5efbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.714097 981 ts_tablet_manager.cc:1918] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.714291 981 log.cc:1199] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b980c746a3c042bfae90aa076acbb1df
I20250623 08:04:34.714686 343 raft_consensus.cc:383] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.714818 981 ts_tablet_manager.cc:1939] T b980c746a3c042bfae90aa076acbb1df P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:34.714964 343 raft_consensus.cc:738] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.715391 343 consensus_queue.cc:260] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.716614 343 ts_tablet_manager.cc:1428] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250623 08:04:34.717288 343 tablet_bootstrap.cc:492] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:34.714444 979 ts_tablet_manager.cc:1918] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.719525 979 log.cc:1199] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/e22f9bae5f7248af99cec06ce6dbbd41
I20250623 08:04:34.719615 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 59015204545e436cb558fe4465a67c97 (DEFAULT_TABLE table=default.Table_e541bb700f4c4e2abc268babf435c6b0 [id=9979c6d8878443c09e6ff9f4aea65d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.720005 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59015204545e436cb558fe4465a67c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.720769 1359 raft_consensus.cc:491] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.721050 1359 raft_consensus.cc:513] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.721329 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 4ea81642050141a9be84f151a5785abc (table default.Table_176ee9ee2de7444ca3c54ee24e3e5667 [id=1475deb3ccfa49d5935900afa29dee6f]) successfully deleted
I20250623 08:04:34.721786 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 55669db7e55042b2bc208ea3218c2d6c (DEFAULT_TABLE table=default.Table_1476eae1111b4f1dbd7c0f5dd0dd9b34 [id=b005275508b748c7a8494ffebdc0bd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.722247 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55669db7e55042b2bc208ea3218c2d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.722195 1359 leader_election.cc:290] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.722921 979 ts_tablet_manager.cc:1939] T e22f9bae5f7248af99cec06ce6dbbd41 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:34.724479 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 78) took 3708 ms. Trace:
I20250623 08:04:34.726282 32492 tablet_service.cc:1468] Processing CreateTablet for tablet c74e8c697d244ac2ac8174bb1fc76e61 (DEFAULT_TABLE table=default.Table_5785d39307cb42758ded4062307c55e3 [id=4ceb1e1117f747b3a5424c01cd47bd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.726642 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c74e8c697d244ac2ac8174bb1fc76e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.727898 1353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Bootstrap starting.
I20250623 08:04:34.733130 1359 raft_consensus.cc:491] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.733403 1359 raft_consensus.cc:513] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.734141 1359 leader_election.cc:290] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.735111 32492 tablet_service.cc:1468] Processing CreateTablet for tablet d5cd316e6f4447be958611ab07ed31fc (DEFAULT_TABLE table=default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac [id=793edd0cb585499197d2f0e57a83be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.735558 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5cd316e6f4447be958611ab07ed31fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.740134 32491 tablet_service.cc:1468] Processing CreateTablet for tablet ea588a48e6164d3db148fdaa306268f6 (DEFAULT_TABLE table=default.Table_b523bec7182443139e891357d295e3a1 [id=1c4298a5f9014d8eb4ccad8c8fbdeeac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.740499 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea588a48e6164d3db148fdaa306268f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.743314 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 696f5d2797ef463bb72a2119501c6ef3 (table default.Table_8a1e4cf5c8a54882b63f1dfd10533986 [id=f9f9b316ab75478ca20caa217c01f3b4]) successfully deleted
I20250623 08:04:34.744467 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 726d80e05a344f27b5a289723cb5aed7 (table default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]) successfully deleted
I20250623 08:04:34.747481 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 55669db7e55042b2bc208ea3218c2d6c (DEFAULT_TABLE table=default.Table_1476eae1111b4f1dbd7c0f5dd0dd9b34 [id=b005275508b748c7a8494ffebdc0bd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.731590 343 tablet_bootstrap.cc:654] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.748107 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55669db7e55042b2bc208ea3218c2d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.746466 979 rpcz_store.cc:276] 0623 08:04:31.016453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.016492 (+ 39us) service_pool.cc:225] Handling call
0623 08:04:34.655225 (+3638733us) ts_tablet_manager.cc:1145] Deleting tablet e22f9bae5f7248af99cec06ce6dbbd41
0623 08:04:34.655237 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.661476 (+ 6239us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.714330 (+ 52854us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.724406 (+ 10076us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96512,"tablet-delete.queue_time_us":3638568}
I20250623 08:04:34.749800 979 tablet_replica.cc:331] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:34.749955 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet e22f9bae5f7248af99cec06ce6dbbd41 (table default.Table_dfc3da325f6849bfba88293a5eabb3e4 [id=ad93708e20ae4ef087d9a9b028d89806]) successfully deleted
I20250623 08:04:34.750149 979 raft_consensus.cc:2241] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.750447 979 raft_consensus.cc:2270] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.751541 979 ts_tablet_manager.cc:1905] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.760211 343 tablet_bootstrap.cc:492] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.760423 343 ts_tablet_manager.cc:1397] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.004s
I20250623 08:04:34.760533 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31ecc74317d449e5b557b07a4fd14895" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.760767 32656 raft_consensus.cc:2466] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.761247 343 raft_consensus.cc:357] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.761507 343 raft_consensus.cc:383] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.761648 343 raft_consensus.cc:738] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.761940 343 consensus_queue.cc:260] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.768219 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:34.772254 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 81) took 2355 ms. Trace:
I20250623 08:04:34.772394 981 rpcz_store.cc:276] 0623 08:04:32.414681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.414728 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:34.536829 (+2122101us) ts_tablet_manager.cc:1145] Deleting tablet b980c746a3c042bfae90aa076acbb1df
0623 08:04:34.536838 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.585966 (+ 49128us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.713956 (+127990us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.769949 (+ 55993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2121782}
I20250623 08:04:34.772701 981 tablet_replica.cc:331] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:34.772878 981 raft_consensus.cc:2241] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:34.773207 981 raft_consensus.cc:2270] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.773766 981 ts_tablet_manager.cc:1905] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.775367 342 tablet_bootstrap.cc:492] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:34.775518 342 ts_tablet_manager.cc:1397] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.248s user 0.007s sys 0.003s
I20250623 08:04:34.776319 342 raft_consensus.cc:357] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.776515 342 raft_consensus.cc:383] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.776607 342 raft_consensus.cc:738] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.777307 339 tablet_bootstrap.cc:492] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:34.777474 339 ts_tablet_manager.cc:1397] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.235s user 0.003s sys 0.000s
I20250623 08:04:34.778179 339 raft_consensus.cc:357] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.778390 339 raft_consensus.cc:383] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.778477 339 raft_consensus.cc:738] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.778714 339 consensus_queue.cc:260] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.779327 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 7dc5e0fbeba4412aa42d116cd8882238 (DEFAULT_TABLE table=default.Table_ea4d110c161f4b008acde5f22ffa3562 [id=79d117a74f18466ab87a1701e641e7da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.779739 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc5e0fbeba4412aa42d116cd8882238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.780932 32055 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6
I20250623 08:04:34.781812 32055 catalog_manager.cc:6652] ReplaceTablet: tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 deleted and replaced by tablet 077bb090b3ea4d029463304d7a6feee5
I20250623 08:04:34.782053 32055 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36116 (request call id 214) took 1114 ms. Trace:
I20250623 08:04:34.782999 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 92005abc61dd40ef9da8124cadc29b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.782163 32055 rpcz_store.cc:276] 0623 08:04:33.667879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.503706 (+835827us) service_pool.cc:225] Handling call
0623 08:04:34.781977 (+278271us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:34.549038 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.549144 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.549151 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:34.549154 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:34.549262 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.549275 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.549282 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:34.549353 (+ 71us) tablet.cc:699] Row locks acquired
0623 08:04:34.549355 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:34.549403 (+ 48us) write_op.cc:270] Start()
0623 08:04:34.549441 (+ 38us) write_op.cc:276] Timestamp: P: 1750665874549352 usec, L: 0
0623 08:04:34.549447 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.549621 (+ 174us) log.cc:844] Serialized 622 byte log entry
0623 08:04:34.779119 (+229498us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.779347 (+ 228us) write_op.cc:301] APPLY: starting
0623 08:04:34.779430 (+ 83us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.779523 (+ 93us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.779580 (+ 57us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.779763 (+ 183us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.779875 (+ 112us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:34.779913 (+ 38us) write_op.cc:312] APPLY: finished
0623 08:04:34.780029 (+ 116us) log.cc:844] Serialized 29 byte log entry
0623 08:04:34.780429 (+ 400us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.780453 (+ 24us) write_op.cc:454] Released schema lock
0623 08:04:34.780461 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:34.781603 (+ 1142us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.781610 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"apply.run_cpu_time_us":733,"apply.run_wall_time_us":1210,"num_ops":2,"prepare.queue_time_us":80,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":675,"raft.queue_time_us":60,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":229635}]]}
I20250623 08:04:34.783905 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 279624b8bb834b95a4f08045968ebfb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.784621 32055 master_service.cc:948] ReplaceTablet: received request to replace tablet e22f9bae5f7248af99cec06ce6dbbd41 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:34.785027 32055 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36116 (request call id 215) took 1086 ms. Trace:
I20250623 08:04:34.785187 32055 rpcz_store.cc:276] 0623 08:04:33.698970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.783945 (+1084975us) service_pool.cc:225] Handling call
0623 08:04:34.784994 (+ 1049us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 08:04:34.786460 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_39a59f2375e743109b067b2de8b6d4db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:34.788182 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 8f83a6d453ec4ffe85282b4385df5f11 (DEFAULT_TABLE table=default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.788658 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f83a6d453ec4ffe85282b4385df5f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.792342 1359 consensus_queue.cc:1035] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:34.792956 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 19536c16ff32460e9db571e4ce411bab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.793573 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e47b10f46b8402e94354ebca573c194 with delete_type TABLET_DATA_DELETED (replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.794292 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 23928d64bcef4c06af81698e688b7f94 (DEFAULT_TABLE table=default.Table_5c6bbeaad8274e05904301c82a72aa74 [id=ea13915f7ffd4e9c9116fb0b12fe0351]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:34.794826 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39a59f2375e743109b067b2de8b6d4db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:34.796267 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 175ee90133994154b7e4b58b0770db3b with delete_type TABLET_DATA_DELETED (replaced by tablet e602d391526d4b0eb0dbf5189afbe075) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.796747 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 with delete_type TABLET_DATA_DELETED (replaced by tablet 205da880685b49b3b8669f73dc2c563b) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.797168 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet f3c850021e914530bb106ab7166be7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.797581 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.798147 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9e53da264a5ca1f70e1846c906f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.798777 343 ts_tablet_manager.cc:1428] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20250623 08:04:34.799185 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23928d64bcef4c06af81698e688b7f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.799163 1278 raft_consensus.cc:491] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.799400 343 tablet_bootstrap.cc:492] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:34.799332 1278 raft_consensus.cc:513] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.800051 1278 leader_election.cc:290] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:34.801662 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:34.801986 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:34.802728 343 tablet_bootstrap.cc:654] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.804790 32453 leader_election.cc:304] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.805249 1366 raft_consensus.cc:2802] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.805399 952 ts_tablet_manager.cc:1918] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:34.805440 1366 raft_consensus.cc:491] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.805625 952 log.cc:1199] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b980c746a3c042bfae90aa076acbb1df
I20250623 08:04:34.805680 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.805675 1366 raft_consensus.cc:3058] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.806111 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 0311ce67989f49e8866234a88e5b4d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.806207 952 ts_tablet_manager.cc:1939] T b980c746a3c042bfae90aa076acbb1df P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:34.806653 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 42) took 3845 ms. Trace:
I20250623 08:04:34.806818 952 rpcz_store.cc:276] 0623 08:04:30.961291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.961334 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:34.591510 (+3630176us) ts_tablet_manager.cc:1145] Deleting tablet b980c746a3c042bfae90aa076acbb1df
0623 08:04:34.591521 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.610002 (+ 18481us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.805264 (+195262us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.806593 (+ 1329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3629981}
I20250623 08:04:34.807015 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ac9f38877b24d619c280a1aaa0b7b08 with delete_type TABLET_DATA_DELETED (replaced by tablet a8de2b1105564fdab386442b1de4cedb) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.807281 952 tablet_replica.cc:331] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:34.807384 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a0b9402a264800a1ea39707c10da0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.807564 952 raft_consensus.cc:2241] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.807775 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet cdec6e9573b041e0a57d1e0e0239d97f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.807801 952 raft_consensus.cc:2270] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.808136 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.808324 952 ts_tablet_manager.cc:1905] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.808539 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.808681 1366 raft_consensus.cc:513] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.808912 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 with delete_type TABLET_DATA_DELETED (replaced by tablet 306c4021b5d94a4996de07c222a8bfd8) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.809423 1366 leader_election.cc:290] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.809442 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 with delete_type TABLET_DATA_DELETED (replaced by tablet db81250d1e164a7d94affd764fc0ff0c) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.809835 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 6926c49b787941d98c647543bd3b9ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.810243 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 52d0a356c74f4daea9093c998b4bf73e with delete_type TABLET_DATA_DELETED (replaced by tablet bb06a340f82047ac8ec3516edb08f4b6) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.810600 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
W20250623 08:04:34.805255 32453 leader_election.cc:343] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 162308da898d4dde8ab5b1ab01d0f393
I20250623 08:04:34.812686 342 consensus_queue.cc:260] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.813434 342 ts_tablet_manager.cc:1428] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20250623 08:04:34.813791 342 tablet_bootstrap.cc:492] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:34.815675 342 tablet_bootstrap.cc:654] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.816497 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.817643 339 ts_tablet_manager.cc:1428] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250623 08:04:34.817994 339 tablet_bootstrap.cc:492] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:34.819723 339 tablet_bootstrap.cc:654] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:34.817075 32589 leader_election.cc:343] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.823555 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31ecc74317d449e5b557b07a4fd14895" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.836457 32655 raft_consensus.cc:3058] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.831308 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet d4221d7500d1458db75926d7b274a654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:34.832419 32358 tablet_service.cc:1468] Processing CreateTablet for tablet de743139c4324faba19ff3e5cfac855d (DEFAULT_TABLE table=default.Table_d3115e0a02304f6d80807841b76504e4 [id=fc36d4cdd9684c6f84517d848ae3395b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.833010 32356 tablet_service.cc:1468] Processing CreateTablet for tablet b61b02cf2904404f8c0ffef7d268fefc (DEFAULT_TABLE table=default.Table_3d6c2aca7a944301bc67f7901cfef8b3 [id=e1d4219a19fc4fffa81e7693469aebce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.840093 32361 tablet_service.cc:1468] Processing CreateTablet for tablet 7dc5e0fbeba4412aa42d116cd8882238 (DEFAULT_TABLE table=default.Table_ea4d110c161f4b008acde5f22ffa3562 [id=79d117a74f18466ab87a1701e641e7da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.840626 32354 tablet_service.cc:1468] Processing CreateTablet for tablet a0b1287cf3b6459eafd032afc77c5824 (DEFAULT_TABLE table=default.Table_cc3b284f368e4123bf1a2da98f4f4fa8 [id=906debc01a6f45a49a86b5f939ac1dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.840986 32360 tablet_service.cc:1468] Processing CreateTablet for tablet 5f957f5575e9475eb9f639e3ad5cb809 (DEFAULT_TABLE table=default.Table_ba8a245f5a564519b522c175b4117f52 [id=4547bcef8b5d4d25acf8de6cbba4f7bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.841358 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 6857437f85ab40f6a24e74ce3b5efbdc (DEFAULT_TABLE table=default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.822506 1054 raft_consensus.cc:491] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.823005 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:34.824312 1366 consensus_queue.cc:1035] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:34.844307 979 ts_tablet_manager.cc:1918] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250623 08:04:34.844651 1054 raft_consensus.cc:513] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.841635 32355 tablet_service.cc:1468] Processing CreateTablet for tablet ea588a48e6164d3db148fdaa306268f6 (DEFAULT_TABLE table=default.Table_b523bec7182443139e891357d295e3a1 [id=1c4298a5f9014d8eb4ccad8c8fbdeeac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.841756 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 8f83a6d453ec4ffe85282b4385df5f11 (DEFAULT_TABLE table=default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.842016 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 55669db7e55042b2bc208ea3218c2d6c (DEFAULT_TABLE table=default.Table_1476eae1111b4f1dbd7c0f5dd0dd9b34 [id=b005275508b748c7a8494ffebdc0bd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.841364 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 23928d64bcef4c06af81698e688b7f94 (DEFAULT_TABLE table=default.Table_5c6bbeaad8274e05904301c82a72aa74 [id=ea13915f7ffd4e9c9116fb0b12fe0351]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.844779 979 log.cc:1199] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/d0ee5a4fe03848229524cc0105226d8c
I20250623 08:04:34.845347 1054 leader_election.cc:290] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:34.824381 32453 consensus_peers.cc:487] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e22f9bae5f7248af99cec06ce6dbbd41. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:34.846905 1054 raft_consensus.cc:491] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.847121 1054 raft_consensus.cc:513] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.847654 1054 leader_election.cc:290] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:34.828771 32043 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.60:39691, user_credentials={real_user=slave}} blocked reactor thread for 50025.1us
I20250623 08:04:34.852792 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:34.854833 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:34.840983 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 59015204545e436cb558fe4465a67c97 (DEFAULT_TABLE table=default.Table_e541bb700f4c4e2abc268babf435c6b0 [id=9979c6d8878443c09e6ff9f4aea65d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.856395 32568 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:34.856868 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:34.857486 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:34.857478 1054 raft_consensus.cc:491] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.857631 1054 raft_consensus.cc:513] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.857793 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:34.858318 1054 leader_election.cc:290] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.862277 1054 raft_consensus.cc:491] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:34.862479 1054 raft_consensus.cc:513] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.863179 1054 leader_election.cc:290] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.863751 342 tablet_bootstrap.cc:492] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:34.863930 342 ts_tablet_manager.cc:1397] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20250623 08:04:34.864693 342 raft_consensus.cc:357] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.864868 342 raft_consensus.cc:383] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.864941 342 raft_consensus.cc:738] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.865198 342 consensus_queue.cc:260] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.865849 342 ts_tablet_manager.cc:1428] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:34.866143 342 tablet_bootstrap.cc:492] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:34.867759 342 tablet_bootstrap.cc:654] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.857993 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:34.868849 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 18ea63263b9b466e922169eecafab747 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]). Expected version 1 got 0
I20250623 08:04:34.869465 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
W20250623 08:04:34.870460 32453 leader_election.cc:343] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 966a159e212040109e2842c15b69fb71
I20250623 08:04:34.870927 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:34.871301 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:34.871582 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:34.872013 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:34.872294 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_cc3ded2eef374e8d9f476cc87bfc2470 [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 3 got 0
I20250623 08:04:34.874171 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:34.874478 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 771ecfc41e2b4007a4d213cec5b128c6 (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:34.874778 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b84b8b8713774ab4896842feb9fb075e (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:34.875147 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_bc1251ce708a4467b5b1c71a1df307e0 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 1 got 0
I20250623 08:04:34.876025 952 ts_tablet_manager.cc:1918] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:34.876760 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:34.877002 952 log.cc:1199] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/79731106915d4cebb9dcc3c10386ffa9
I20250623 08:04:34.877557 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:34.877986 952 ts_tablet_manager.cc:1939] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:34.878676 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 39) took 3919 ms. Trace:
I20250623 08:04:34.879492 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 136e482995a24c5daec55a701f3091fa (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:34.880134 952 rpcz_store.cc:276] 0623 08:04:30.959559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.961559 (+ 2000us) service_pool.cc:225] Handling call
0623 08:04:34.807188 (+3845629us) ts_tablet_manager.cc:1145] Deleting tablet 79731106915d4cebb9dcc3c10386ffa9
0623 08:04:34.807201 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.859932 (+ 52731us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.875853 (+ 15921us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.878638 (+ 2785us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3845368}
I20250623 08:04:34.881456 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 43) took 3919 ms. Trace:
I20250623 08:04:34.881757 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:34.882120 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_049497d4290c4bdfaf1e609b8b2ac9ac [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 2 got 0
I20250623 08:04:34.881076 342 tablet_bootstrap.cc:492] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
W20250623 08:04:34.866829 32589 leader_election.cc:343] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.882789 342 ts_tablet_manager.cc:1397] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250623 08:04:34.882794 32589 leader_election.cc:304] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.869369 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1c82e4d2db64026a04c5c84bc1e09ec" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.883548 1278 raft_consensus.cc:2747] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:34.883612 32654 raft_consensus.cc:2466] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.883603 342 raft_consensus.cc:357] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.883733 1278 raft_consensus.cc:491] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.883759 342 raft_consensus.cc:383] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.883852 342 raft_consensus.cc:738] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.883854 1278 raft_consensus.cc:513] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.872618 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffafd10f26740c9a5b6c9bdc3bb23a0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.884104 32649 raft_consensus.cc:2466] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.884073 342 consensus_queue.cc:260] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.872640 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6ed4cb252b1452087f8ee53ec78d985" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:34.884449 1278 leader_election.cc:290] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:34.884536 32653 raft_consensus.cc:2466] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.884660 342 ts_tablet_manager.cc:1428] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:34.876199 32703 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:34.873821 343 tablet_bootstrap.cc:492] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.884897 343 ts_tablet_manager.cc:1397] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.086s user 0.000s sys 0.005s
I20250623 08:04:34.885629 343 raft_consensus.cc:357] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.885823 343 raft_consensus.cc:383] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.885906 343 raft_consensus.cc:738] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.886139 343 consensus_queue.cc:260] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.886498 32453 leader_election.cc:304] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.886607 343 ts_tablet_manager.cc:1428] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:34.886801 32453 leader_election.cc:304] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.886906 343 tablet_bootstrap.cc:492] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:34.887065 32453 leader_election.cc:304] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.887224 1366 raft_consensus.cc:2802] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.887315 1366 raft_consensus.cc:491] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.887414 1366 raft_consensus.cc:3058] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.887482 1159 raft_consensus.cc:2802] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.887472 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 077bb090b3ea4d029463304d7a6feee5) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:34.887585 1159 raft_consensus.cc:491] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.887679 1159 raft_consensus.cc:3058] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.888916 343 tablet_bootstrap.cc:654] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.890038 342 tablet_bootstrap.cc:492] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:34.866803 32655 raft_consensus.cc:2466] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:34.890619 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a414144276f6484ea4690204752e2142" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:34.890743 1054 raft_consensus.cc:2802] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:34.890859 1054 raft_consensus.cc:491] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:34.890960 1054 raft_consensus.cc:3058] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.891450 32453 leader_election.cc:304] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.891711 848 raft_consensus.cc:2802] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.891857 848 raft_consensus.cc:695] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:34.892138 848 consensus_queue.cc:237] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:34.894306 32589 leader_election.cc:343] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.895126 1366 raft_consensus.cc:513] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.889652 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b980c746a3c042bfae90aa076acbb1df (table default.Table_71cce1d280bf41b7bae2f47aeec9a39e [id=09707632913546819a7f78d5d6bc5b2a]) successfully deleted
I20250623 08:04:34.897526 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de743139c4324faba19ff3e5cfac855d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.897818 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea588a48e6164d3db148fdaa306268f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.898576 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 077bb090b3ea4d029463304d7a6feee5) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:34.899947 32347 tablet_service.cc:1468] Processing CreateTablet for tablet c74e8c697d244ac2ac8174bb1fc76e61 (DEFAULT_TABLE table=default.Table_5785d39307cb42758ded4062307c55e3 [id=4ceb1e1117f747b3a5424c01cd47bd6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.900787 1054 raft_consensus.cc:513] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.901650 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a414144276f6484ea4690204752e2142" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.902117 32350 tablet_service.cc:1468] Processing CreateTablet for tablet d5cd316e6f4447be958611ab07ed31fc (DEFAULT_TABLE table=default.Table_e6f7f54745c24f1bb28f268b6ffcc0ac [id=793edd0cb585499197d2f0e57a83be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:34.902479 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1c82e4d2db64026a04c5c84bc1e09ec" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.902681 32653 raft_consensus.cc:3058] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.903023 952 rpcz_store.cc:276] 0623 08:04:30.961664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.961711 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:34.881283 (+3919572us) ts_tablet_manager.cc:1145] Deleting tablet a414144276f6484ea4690204752e2142
0623 08:04:34.881291 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.881411 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3919374}
I20250623 08:04:34.903393 952 tablet_replica.cc:331] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:34.903591 952 raft_consensus.cc:2241] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:34.903743 952 raft_consensus.cc:2270] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:34.903384 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6ed4cb252b1452087f8ee53ec78d985" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.904275 952 ts_tablet_manager.cc:1905] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:34.904408 32655 raft_consensus.cc:3058] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.905594 32653 raft_consensus.cc:2466] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:34.906042 32453 leader_election.cc:304] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:34.906360 1359 raft_consensus.cc:2802] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:34.906574 1359 raft_consensus.cc:695] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
W20250623 08:04:34.906733 32589 leader_election.cc:343] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:34.906870 1359 consensus_queue.cc:237] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.907024 32589 leader_election.cc:304] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:34.896126 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:34.907333 1278 raft_consensus.cc:2747] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:34.907457 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_32db9f68dd974fb8aff372a8eeccce3d [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 1 got 0
I20250623 08:04:34.907704 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:34.907716 1159 raft_consensus.cc:513] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.907859 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f83f68ebda7e4826bbb17278cd4ac5a8 (table default.Table_6acb69304c424830a635f22f12343b31 [id=c1782341d0d34662889cff00d557ff81]). Expected version 1 got 0
I20250623 08:04:34.908324 1159 leader_election.cc:290] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.908666 342 tablet_bootstrap.cc:654] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:34.909271 1054 leader_election.cc:290] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.910095 1366 leader_election.cc:290] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:34.910627 32059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db81250d1e164a7d94affd764fc0ff0c consensus state has no leader
I20250623 08:04:34.910862 32059 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 211) took 1281 ms. Trace:
I20250623 08:04:34.910953 32059 rpcz_store.cc:276] 0623 08:04:33.628853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.420997 (+792144us) service_pool.cc:225] Handling call
0623 08:04:34.458272 (+ 37275us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:34.458875 (+ 603us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:34.459536 (+ 661us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:34.459666 (+ 130us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:34.501618 (+ 41952us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:34.910779 (+409161us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:34.460290 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.460395 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.460402 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:34.460405 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:34.460503 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.460515 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.460523 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:34.460586 (+ 63us) tablet.cc:699] Row locks acquired
0623 08:04:34.460588 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:34.460624 (+ 36us) write_op.cc:270] Start()
0623 08:04:34.460654 (+ 30us) write_op.cc:276] Timestamp: P: 1750665874460585 usec, L: 0
0623 08:04:34.460660 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.460817 (+ 157us) log.cc:844] Serialized 405 byte log entry
0623 08:04:34.500017 (+ 39200us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.500703 (+ 686us) write_op.cc:301] APPLY: starting
0623 08:04:34.500717 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.500756 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.500798 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.500837 (+ 39us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.500872 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:34.500882 (+ 10us) write_op.cc:312] APPLY: finished
0623 08:04:34.500944 (+ 62us) log.cc:844] Serialized 23 byte log entry
0623 08:04:34.500997 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.501011 (+ 14us) write_op.cc:454] Released schema lock
0623 08:04:34.501018 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":633,"apply.run_cpu_time_us":377,"apply.run_wall_time_us":377,"num_ops":1,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":624,"prepare.run_wall_time_us":625,"raft.queue_time_us":98,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":39315}]]}
I20250623 08:04:34.911268 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffafd10f26740c9a5b6c9bdc3bb23a0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:34.911851 32653 raft_consensus.cc:3058] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.919957 979 ts_tablet_manager.cc:1939] T d0ee5a4fe03848229524cc0105226d8c P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:34.924714 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712abfca1fa4489f882534c33b9db1be" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.924376 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 61) took 3914 ms. Trace:
I20250623 08:04:34.925295 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f957f5575e9475eb9f639e3ad5cb809. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.926537 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0b1287cf3b6459eafd032afc77c5824. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.929358 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c74e8c697d244ac2ac8174bb1fc76e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.930244 32428 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:34.930528 32376 raft_consensus.cc:3058] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:34.934940 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9c8d480b8ba49a1be140b0e0a79f4e4" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.935070 848 raft_consensus.cc:491] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.935205 848 raft_consensus.cc:513] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.935685 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5cd316e6f4447be958611ab07ed31fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.935910 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6857437f85ab40f6a24e74ce3b5efbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.936635 848 leader_election.cc:290] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.937641 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7adec1482119405daca4aa603770ae22" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.939087 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712abfca1fa4489f882534c33b9db1be" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.938180 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7adec1482119405daca4aa603770ae22" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.941520 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1c82e4d2db64026a04c5c84bc1e09ec" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.942040 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31ecc74317d449e5b557b07a4fd14895" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.941133 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9c8d480b8ba49a1be140b0e0a79f4e4" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.942694 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f83a6d453ec4ffe85282b4385df5f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.941805 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31ecc74317d449e5b557b07a4fd14895" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.947086 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dc696236a15420e9666cca708068fb2" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.946935 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "162308da898d4dde8ab5b1ab01d0f393" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.951777 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6ed4cb252b1452087f8ee53ec78d985" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.952311 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffafd10f26740c9a5b6c9bdc3bb23a0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.952661 339 tablet_bootstrap.cc:492] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:34.952881 339 ts_tablet_manager.cc:1397] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.135s user 0.007s sys 0.000s
I20250623 08:04:34.953111 848 raft_consensus.cc:491] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:34.953275 848 raft_consensus.cc:513] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.953909 848 leader_election.cc:290] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:34.953984 339 raft_consensus.cc:357] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.954190 339 raft_consensus.cc:383] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:34.954289 339 raft_consensus.cc:738] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:34.955060 339 consensus_queue.cc:260] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.958778 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59015204545e436cb558fe4465a67c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.960310 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23928d64bcef4c06af81698e688b7f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:34.960795 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1c82e4d2db64026a04c5c84bc1e09ec" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.961621 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6ed4cb252b1452087f8ee53ec78d985" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.962548 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffafd10f26740c9a5b6c9bdc3bb23a0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:34.962813 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "966a159e212040109e2842c15b69fb71" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.964758 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdc9d47f2232403fa71e14a6fe6f33c3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.965314 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a0b9402a264800a1ea39707c10da0c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:34.983578 1353 log.cc:826] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Log is configured to *not* fsync() on all Append() calls
W20250623 08:04:34.999858 32453 leader_election.cc:343] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.000043 32453 leader_election.cc:304] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
W20250623 08:04:35.000226 32453 raft_consensus.cc:2684] T 79731106915d4cebb9dcc3c10386ffa9 P 59f7c614254f42009306b13337f06576: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:35.006994 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "358cb6ac24414bdfbc32155e732f9ac8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:35.007774 32453 consensus_peers.cc:487] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:35.008018 32453 consensus_peers.cc:487] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:35.008430 32453 leader_election.cc:343] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.015065 32453 leader_election.cc:343] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.015714 32453 consensus_peers.cc:487] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 175ee90133994154b7e4b58b0770db3b. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:35.016163 32453 leader_election.cc:343] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.017131 32453 consensus_peers.cc:487] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:35.017397 32453 leader_election.cc:343] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.018543 32453 consensus_peers.cc:487] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:35.019096 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55669db7e55042b2bc208ea3218c2d6c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:35.021389 32453 leader_election.cc:343] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.021836 32453 leader_election.cc:343] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.022166 32453 leader_election.cc:343] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.023288 32453 leader_election.cc:343] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.023653 32453 leader_election.cc:343] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.024000 32453 leader_election.cc:343] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.024322 32453 leader_election.cc:343] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.024498 32453 leader_election.cc:304] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.024863 1159 raft_consensus.cc:2747] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:35.025322 32453 leader_election.cc:343] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.026005 32453 leader_election.cc:343] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.026660 32453 leader_election.cc:343] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.028337 32453 leader_election.cc:343] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.029266 32453 leader_election.cc:343] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.031514 1332 raft_consensus.cc:491] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:35.036198 1332 raft_consensus.cc:513] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:35.036767 339 ts_tablet_manager.cc:1428] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.066s user 0.003s sys 0.000s
I20250623 08:04:35.037142 339 tablet_bootstrap.cc:492] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:35.047163 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc5e0fbeba4412aa42d116cd8882238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:35.050601 1332 leader_election.cc:290] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.057725 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 79731106915d4cebb9dcc3c10386ffa9 (table default.Table_d44a2132435f43e7847bfa349645a605 [id=b42459e9ca5e475cb5083d68bb6349be]) successfully deleted
W20250623 08:04:35.058040 32043 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 140209us
I20250623 08:04:34.936954 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a0b9402a264800a1ea39707c10da0c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.058990 32649 raft_consensus.cc:2466] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:35.063063 32314 consensus_peers.cc:487] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9918b15e48144ef0b8f3aedc2135be59. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:34.975670 343 tablet_bootstrap.cc:492] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:34.954686 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdc9d47f2232403fa71e14a6fe6f33c3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.063570 32654 raft_consensus.cc:2466] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:34.987898 1278 raft_consensus.cc:491] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.063789 1278 raft_consensus.cc:513] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:34.949175 32653 raft_consensus.cc:2466] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.064466 1278 leader_election.cc:290] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:35.043083 1128 raft_consensus.cc:491] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.064966 1128 raft_consensus.cc:513] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.065543 1128 leader_election.cc:290] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:35.070154 1359 raft_consensus.cc:491] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:35.074816 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc95f12ab5d4b9b9f0729cb88aabbaa" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.063566 343 ts_tablet_manager.cc:1397] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.177s user 0.005s sys 0.018s
I20250623 08:04:35.030304 32453 leader_election.cc:304] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.079989 1159 raft_consensus.cc:2747] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 08:04:35.080072 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 50142.3us
W20250623 08:04:35.080422 32453 leader_election.cc:343] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.080641 32453 leader_election.cc:343] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.080636 343 raft_consensus.cc:357] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.080825 343 raft_consensus.cc:383] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.080922 343 raft_consensus.cc:738] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.081192 343 consensus_queue.cc:260] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.081779 343 ts_tablet_manager.cc:1428] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250623 08:04:35.081892 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet a414144276f6484ea4690204752e2142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a414144276f6484ea4690204752e2142 already in progress: creating tablet
I20250623 08:04:35.082082 343 tablet_bootstrap.cc:492] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.070643 1359 raft_consensus.cc:513] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:35.084030 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 32c82141edea4068b54755a85c6d7fb8 (table default.Table_c30b48097bd643b18b2a744839780b9e [id=7e59ad2235c1418d97f2b73ff2a48300]) successfully deleted
I20250623 08:04:35.056797 32655 raft_consensus.cc:2466] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.000926 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "358cb6ac24414bdfbc32155e732f9ac8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.085388 848 consensus_queue.cc:1035] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:35.085563 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 79731106915d4cebb9dcc3c10386ffa9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79731106915d4cebb9dcc3c10386ffa9 already in progress: creating tablet
I20250623 08:04:35.085794 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet b980c746a3c042bfae90aa076acbb1df (table default.Table_71cce1d280bf41b7bae2f47aeec9a39e [id=09707632913546819a7f78d5d6bc5b2a]) successfully deleted
W20250623 08:04:35.086813 32053 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 736eb56dae194df2af90bf33d3294a4e consensus state has no leader
I20250623 08:04:35.090484 32513 raft_consensus.cc:2466] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
W20250623 08:04:35.090811 32056 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5cd316e6f4447be958611ab07ed31fc consensus state has no leader
I20250623 08:04:35.091215 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c33fa0a7ff34d2985c5e07fcf88eaa1" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.091246 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c6f98e2db1a4b0789c5819b2c7d4bb6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:35.091734 32589 leader_election.cc:343] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.091854 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b61b02cf2904404f8c0ffef7d268fefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:35.092762 1332 raft_consensus.cc:491] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 08:04:35.093140 32589 leader_election.cc:343] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.093422 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 36c8714cfbd84dd38cf5c0f305ef6843
I20250623 08:04:35.094192 1359 leader_election.cc:290] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:35.094417 1332 raft_consensus.cc:513] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.094981 1332 leader_election.cc:290] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:35.095471 32314 consensus_peers.cc:487] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 726d80e05a344f27b5a289723cb5aed7. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:35.096877 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c33fa0a7ff34d2985c5e07fcf88eaa1" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.095570 32054 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 212) took 1442 ms. Trace:
W20250623 08:04:35.102679 32052 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:39129 (request call id 170) took 2015 ms (client timeout 1000 ms). Trace:
I20250623 08:04:35.103693 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 36c8714cfbd84dd38cf5c0f305ef6843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:34 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.104274 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 with delete_type TABLET_DATA_DELETED (replaced by tablet 306c4021b5d94a4996de07c222a8bfd8) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.104527 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet d4221d7500d1458db75926d7b274a654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.104743 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet 205da880685b49b3b8669f73dc2c563b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.104943 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet cde7df7d39a54114a4dcb28cfd1eb429 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f1fb8231e8e45208ac4f412134d8696) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.106271 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 696f5d2797ef463bb72a2119501c6ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:35.102804 32052 rpcz_store.cc:269] 0623 08:04:33.083826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.299357 (+215531us) service_pool.cc:225] Handling call
0623 08:04:35.099458 (+1800101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:33.784189 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:33.784290 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:33.784296 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:33.784298 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:33.784424 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:33.784432 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:33.784439 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:33.784528 (+ 89us) tablet.cc:699] Row locks acquired
0623 08:04:33.784530 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:33.784561 (+ 31us) write_op.cc:270] Start()
0623 08:04:33.784589 (+ 28us) write_op.cc:276] Timestamp: P: 1750665873784542 usec, L: 0
0623 08:04:33.784592 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:33.784737 (+ 145us) log.cc:844] Serialized 901 byte log entry
0623 08:04:33.929040 (+144303us) op_driver.cc:464] REPLICATION: finished
0623 08:04:33.929760 (+ 720us) write_op.cc:301] APPLY: starting
0623 08:04:33.929770 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:33.929803 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:33.929851 (+ 48us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:33.929921 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:33.929960 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:33.929971 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:33.930040 (+ 69us) log.cc:844] Serialized 29 byte log entry
0623 08:04:33.930091 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.930111 (+ 20us) write_op.cc:454] Released schema lock
0623 08:04:33.930118 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:33.930183 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.930186 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:33.936568 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:33.936666 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:33.936672 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:33.936674 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:33.936755 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:33.936763 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:33.936766 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:33.936813 (+ 47us) tablet.cc:699] Row locks acquired
0623 08:04:33.936815 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:33.936844 (+ 29us) write_op.cc:270] Start()
0623 08:04:33.936876 (+ 32us) write_op.cc:276] Timestamp: P: 1750665873936808 usec, L: 0
0623 08:04:33.936881 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:33.937030 (+ 149us) log.cc:844] Serialized 356 byte log entry
0623 08:04:33.954238 (+ 17208us) op_driver.cc:464] REPLICATION: finished
0623 08:04:33.955154 (+ 916us) write_op.cc:301] APPLY: starting
0623 08:04:33.955166 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:33.955206 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:33.955244 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:33.955290 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:33.955324 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:33.955334 (+ 10us) write_op.cc:312] APPLY: finished
0623 08:04:33.955404 (+ 70us) log.cc:844] Serialized 23 byte log entry
0623 08:04:33.955449 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.955464 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:33.955471 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:33.955540 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.955544 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:34.374598 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.374759 (+ 161us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.374767 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:34.374772 (+ 5us) tablet.cc:660] Decoding operations
0623 08:04:34.374899 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.374914 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.374921 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:34.375000 (+ 79us) tablet.cc:699] Row locks acquired
0623 08:04:34.375004 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:34.375043 (+ 39us) write_op.cc:270] Start()
0623 08:04:34.375084 (+ 41us) write_op.cc:276] Timestamp: P: 1750665874375012 usec, L: 0
0623 08:04:34.375090 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.375280 (+ 190us) log.cc:844] Serialized 356 byte log entry
0623 08:04:34.391462 (+ 16182us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.391638 (+ 176us) write_op.cc:301] APPLY: starting
0623 08:04:34.391658 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.391703 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.391727 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.391791 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.391836 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:34.391847 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:34.391936 (+ 89us) log.cc:844] Serialized 23 byte log entry
0623 08:04:34.392004 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.392026 (+ 22us) write_op.cc:454] Released schema lock
0623 08:04:34.392035 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:34.392116 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.392122 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":41731,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":684,"apply.run_cpu_time_us":418,"apply.run_wall_time_us":418,"num_ops":2,"prepare.queue_time_us":1628,"prepare.run_cpu_time_us":638,"prepare.run_wall_time_us":638,"raft.queue_time_us":60,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":144422}],["op",{"apply.queue_time_us":874,"apply.run_cpu_time_us":381,"apply.run_wall_time_us":382,"num_ops":1,"prepare.queue_time_us":4622,"prepare.run_cpu_time_us":544,"prepare.run_wall_time_us":543,"raft.queue_time_us":323,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":17335}],["op",{"apply.queue_time_us":114,"apply.run_cpu_time_us":466,"apply.run_wall_time_us":466,"num_ops":1,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":794,"prepare.run_wall_time_us":794,"raft.queue_time_us":71,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":16334}]]}
I20250623 08:04:35.109822 32453 leader_election.cc:304] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.110507 32453 leader_election.cc:304] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.110924 1159 raft_consensus.cc:2802] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.111027 1159 raft_consensus.cc:491] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.111080 32453 leader_election.cc:304] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.111147 1159 raft_consensus.cc:3058] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.112772 1128 raft_consensus.cc:491] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.112954 1128 raft_consensus.cc:513] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.113050 848 raft_consensus.cc:2802] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.113471 848 raft_consensus.cc:491] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.113616 848 raft_consensus.cc:3058] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.113830 1128 leader_election.cc:290] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:35.114782 1359 raft_consensus.cc:2802] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.115190 1359 raft_consensus.cc:695] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.117607 32453 leader_election.cc:304] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.117970 1366 raft_consensus.cc:2802] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.118377 952 ts_tablet_manager.cc:1918] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.118614 1366 raft_consensus.cc:695] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.118551 1359 consensus_queue.cc:237] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.119972 343 tablet_bootstrap.cc:654] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.125136 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_68d8428833794f57b663cdae5b7d6cb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:35.093349 32050 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b] alter complete (version 1)
W20250623 08:04:35.126433 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68d8428833794f57b663cdae5b7d6cb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:35.114710 32053 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 210) took 1473 ms. Trace:
I20250623 08:04:35.131283 32053 rpcz_store.cc:276] 0623 08:04:33.628706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.419589 (+790883us) service_pool.cc:225] Handling call
0623 08:04:34.420041 (+ 452us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:34.420272 (+ 231us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:34.457564 (+ 37292us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:34.457647 (+ 83us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:34.501515 (+ 43868us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:35.102005 (+600490us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:34.458262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.458392 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.458398 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:34.458401 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:34.458519 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.458533 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.458540 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:34.458615 (+ 75us) tablet.cc:699] Row locks acquired
0623 08:04:34.458619 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:34.458660 (+ 41us) write_op.cc:270] Start()
0623 08:04:34.458694 (+ 34us) write_op.cc:276] Timestamp: P: 1750665874458620 usec, L: 0
0623 08:04:34.458697 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.458912 (+ 215us) log.cc:844] Serialized 405 byte log entry
0623 08:04:34.498145 (+ 39233us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.500126 (+ 1981us) write_op.cc:301] APPLY: starting
0623 08:04:34.500145 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.500188 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.500221 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.500280 (+ 59us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.500322 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:34.500334 (+ 12us) write_op.cc:312] APPLY: finished
0623 08:04:34.500430 (+ 96us) log.cc:844] Serialized 23 byte log entry
0623 08:04:34.500493 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.500513 (+ 20us) write_op.cc:454] Released schema lock
0623 08:04:34.500521 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:34.500604 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.500610 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":15046,"child_traces":[["op",{"apply.queue_time_us":1919,"apply.run_cpu_time_us":469,"apply.run_wall_time_us":468,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":764,"prepare.run_wall_time_us":764,"raft.queue_time_us":346,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":39411}]]}
I20250623 08:04:35.132071 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_d74a4572f91e48d0b1c123e11ecf5b51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:35.132617 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d74a4572f91e48d0b1c123e11ecf5b51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:34.925252 979 rpcz_store.cc:276] 0623 08:04:31.009261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.016712 (+ 7451us) service_pool.cc:225] Handling call
0623 08:04:34.749690 (+3732978us) ts_tablet_manager.cc:1145] Deleting tablet d0ee5a4fe03848229524cc0105226d8c
0623 08:04:34.749704 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.826236 (+ 76532us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.843862 (+ 17626us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:34.924136 (+ 80274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3732786}
I20250623 08:04:35.133963 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 077bb090b3ea4d029463304d7a6feee5) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:35.114841 32055 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 216) took 1387 ms. Trace:
I20250623 08:04:35.134435 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 60) took 4125 ms. Trace:
I20250623 08:04:35.134591 32055 rpcz_store.cc:276] 0623 08:04:33.713709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.786270 (+1072561us) service_pool.cc:225] Handling call
0623 08:04:34.794991 (+ 8721us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:34.795568 (+ 577us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.101269 (+305701us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.101373 (+ 104us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.101468 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:34.796276 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.796432 (+ 156us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.796439 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:34.796441 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:34.796542 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.796558 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.796564 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:34.796656 (+ 92us) tablet.cc:699] Row locks acquired
0623 08:04:34.796659 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:34.796699 (+ 40us) write_op.cc:270] Start()
0623 08:04:34.796736 (+ 37us) write_op.cc:276] Timestamp: P: 1750665874796680 usec, L: 0
0623 08:04:34.796741 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.796939 (+ 198us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.099937 (+302998us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.100253 (+ 316us) write_op.cc:301] APPLY: starting
0623 08:04:35.100270 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.100317 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.100344 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.100440 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.100477 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.100485 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:35.100557 (+ 72us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.100637 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.100656 (+ 19us) write_op.cc:454] Released schema lock
0623 08:04:35.100663 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.100740 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.100744 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":252,"apply.run_cpu_time_us":486,"apply.run_wall_time_us":485,"num_ops":2,"prepare.queue_time_us":95,"prepare.run_cpu_time_us":743,"prepare.run_wall_time_us":743,"replication_time_us":303160}]]}
I20250623 08:04:35.134884 979 rpcz_store.cc:276] 0623 08:04:31.009141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.016923 (+ 7782us) service_pool.cc:225] Handling call
0623 08:04:35.133921 (+4116998us) ts_tablet_manager.cc:1145] Deleting tablet 4fe30079045d437aa61bb5bfbfb96133
0623 08:04:35.133933 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.134303 (+ 370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4116741}
I20250623 08:04:35.135507 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_5afa01a4e2fa4b7c97e51a681da648d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:35.136057 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5afa01a4e2fa4b7c97e51a681da648d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:35.136164 979 tablet_replica.cc:331] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:35.114907 32056 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 209) took 1470 ms. Trace:
I20250623 08:04:35.115347 32054 rpcz_store.cc:276] 0623 08:04:33.651711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.440227 (+788516us) service_pool.cc:225] Handling call
0623 08:04:34.459292 (+ 19065us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:34.459477 (+ 185us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:34.459537 (+ 60us) catalog_manager.cc:2799] Locking tablets
0623 08:04:34.544978 (+ 85441us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:34.746265 (+201287us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:34.746417 (+ 152us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:34.918528 (+172111us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:35.093889 (+175361us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:34.545932 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.546053 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.546059 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:34.546061 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:34.546234 (+ 173us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.546248 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.546252 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:34.546347 (+ 95us) tablet.cc:699] Row locks acquired
0623 08:04:34.546350 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:34.546395 (+ 45us) write_op.cc:270] Start()
0623 08:04:34.546428 (+ 33us) write_op.cc:276] Timestamp: P: 1750665874546347 usec, L: 0
0623 08:04:34.546433 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.546645 (+ 212us) log.cc:844] Serialized 725 byte log entry
0623 08:04:34.745271 (+198626us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.745415 (+ 144us) write_op.cc:301] APPLY: starting
0623 08:04:34.745434 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.745479 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.745511 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.745581 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.745619 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:34.745626 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:34.745710 (+ 84us) log.cc:844] Serialized 29 byte log entry
0623 08:04:34.745771 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.745793 (+ 22us) write_op.cc:454] Released schema lock
0623 08:04:34.745814 (+ 21us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:34.745881 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.745886 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":459,"apply.run_wall_time_us":459,"num_ops":2,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":814,"prepare.run_wall_time_us":814,"raft.queue_time_us":649,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":310,"replication_time_us":198797}]]}
I20250623 08:04:35.137022 32054 master_service.cc:948] ReplaceTablet: received request to replace tablet 501b5233522f43c7aed3c9770adb5c03 from {username='slave'} at 127.0.0.1:36116
W20250623 08:04:35.123164 32058 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:39501 (request call id 166) took 2858 ms (client timeout 1000 ms). Trace:
I20250623 08:04:35.136888 32056 rpcz_store.cc:276] 0623 08:04:33.623542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.415207 (+791665us) service_pool.cc:225] Handling call
0623 08:04:34.415669 (+ 462us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:34.415900 (+ 231us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:34.416125 (+ 225us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:34.416207 (+ 82us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:34.478280 (+ 62073us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:35.094141 (+615861us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:34.421986 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.422123 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.422246 (+ 123us) write_op.cc:435] Acquired schema lock
0623 08:04:34.422249 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:34.422496 (+ 247us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.422511 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.422516 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:34.422581 (+ 65us) tablet.cc:699] Row locks acquired
0623 08:04:34.422583 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:34.422622 (+ 39us) write_op.cc:270] Start()
0623 08:04:34.422667 (+ 45us) write_op.cc:276] Timestamp: P: 1750665874422587 usec, L: 0
0623 08:04:34.422670 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.422872 (+ 202us) log.cc:844] Serialized 405 byte log entry
0623 08:04:34.476665 (+ 53793us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.476936 (+ 271us) write_op.cc:301] APPLY: starting
0623 08:04:34.476957 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.477010 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.477044 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.477121 (+ 77us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.477179 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:34.477190 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:34.477292 (+ 102us) log.cc:844] Serialized 23 byte log entry
0623 08:04:34.477364 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.477405 (+ 41us) write_op.cc:454] Released schema lock
0623 08:04:34.477414 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:34.477498 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.477504 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":210,"apply.run_cpu_time_us":549,"apply.run_wall_time_us":549,"num_ops":1,"prepare.queue_time_us":5315,"prepare.run_cpu_time_us":775,"prepare.run_wall_time_us":1301,"raft.queue_time_us":376,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":53954}]]}
W20250623 08:04:35.137259 32058 rpcz_store.cc:269] 0623 08:04:32.265071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.449237 (+184166us) service_pool.cc:225] Handling call
0623 08:04:33.277361 (+828124us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.295983 (+ 18622us) write_op.cc:454] Released schema lock
0623 08:04:35.123093 (+1827110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:33.230198 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:33.230337 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:33.230345 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:33.230347 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:33.230554 (+ 207us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:33.230571 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:33.230576 (+ 5us) tablet.cc:683] Acquiring locks for 4 operations
0623 08:04:33.230704 (+ 128us) tablet.cc:699] Row locks acquired
0623 08:04:33.230707 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:33.230815 (+ 108us) write_op.cc:270] Start()
0623 08:04:33.230852 (+ 37us) write_op.cc:276] Timestamp: P: 1750665873230677 usec, L: 0
0623 08:04:33.230856 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:33.231118 (+ 262us) log.cc:844] Serialized 1650 byte log entry
0623 08:04:33.245257 (+ 14139us) op_driver.cc:464] REPLICATION: finished
0623 08:04:33.245411 (+ 154us) write_op.cc:301] APPLY: starting
0623 08:04:33.245430 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:33.245526 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:33.245555 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:33.245657 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:33.245698 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0623 08:04:33.245707 (+ 9us) write_op.cc:312] APPLY: finished
0623 08:04:33.245806 (+ 99us) log.cc:844] Serialized 41 byte log entry
0623 08:04:33.245870 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:33.245896 (+ 26us) write_op.cc:454] Released schema lock
0623 08:04:33.245903 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":11469,"reported_tablets":115,"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":547,"apply.run_wall_time_us":546,"num_ops":4,"prepare.queue_time_us":6646,"prepare.run_cpu_time_us":1031,"prepare.run_wall_time_us":1068,"raft.queue_time_us":425,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":267,"replication_time_us":14361,"spinlock_wait_cycles":1280}]]}
I20250623 08:04:35.138154 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_57b13d90b3894f80b5a1cec576d2fb3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:35.138687 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57b13d90b3894f80b5a1cec576d2fb3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:35.138693 32058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_06b731510ae743608f1fbc4d250acf14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:35.139453 32058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06b731510ae743608f1fbc4d250acf14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:35.140065 979 raft_consensus.cc:2241] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:35.141335 979 raft_consensus.cc:2270] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.146454 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c6f98e2db1a4b0789c5819b2c7d4bb6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.118873 1366 consensus_queue.cc:237] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.146939 952 log.cc:1199] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/67c44697ecab4dd9b43f544a602897ca
I20250623 08:04:35.150013 952 ts_tablet_manager.cc:1939] T 67c44697ecab4dd9b43f544a602897ca P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:35.150784 979 ts_tablet_manager.cc:1905] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.151216 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 44) took 4188 ms. Trace:
I20250623 08:04:35.151356 952 rpcz_store.cc:276] 0623 08:04:30.962013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.962100 (+ 87us) service_pool.cc:225] Handling call
0623 08:04:34.903319 (+3941219us) ts_tablet_manager.cc:1145] Deleting tablet 67c44697ecab4dd9b43f544a602897ca
0623 08:04:34.903330 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:34.924053 (+ 20723us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.112804 (+188751us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.150289 (+ 37485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3941015}
I20250623 08:04:35.151702 952 tablet_replica.cc:331] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:35.151959 952 raft_consensus.cc:2241] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.152231 952 pending_rounds.cc:70] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:35.152297 952 pending_rounds.cc:77] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727422007963648 op_type: NO_OP noop_request { }
I20250623 08:04:35.152456 952 raft_consensus.cc:2887] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:35.152575 952 raft_consensus.cc:2270] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.153398 952 ts_tablet_manager.cc:1905] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.164223 343 tablet_bootstrap.cc:492] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:35.164789 343 ts_tablet_manager.cc:1397] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.083s user 0.006s sys 0.005s
I20250623 08:04:35.166087 343 raft_consensus.cc:357] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.166378 343 raft_consensus.cc:383] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.166481 343 raft_consensus.cc:738] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.166812 343 consensus_queue.cc:260] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.167527 343 ts_tablet_manager.cc:1428] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:35.168036 343 tablet_bootstrap.cc:492] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.170030 343 tablet_bootstrap.cc:654] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.171823 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef663ffdaf141aea5e6269b6b2e1c89" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.171924 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
W20250623 08:04:35.175380 32314 consensus_peers.cc:487] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:35.175503 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 79713.7us
I20250623 08:04:35.179503 342 tablet_bootstrap.cc:492] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
W20250623 08:04:35.172772 32589 leader_election.cc:343] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.174937 1159 raft_consensus.cc:513] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.181980 339 tablet_bootstrap.cc:654] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.182979 32314 leader_election.cc:304] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:35.183293 1332 raft_consensus.cc:2802] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:35.183383 1332 raft_consensus.cc:491] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:35.183487 1332 raft_consensus.cc:3058] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:35.186146 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38ecd4c243b840f4b66d8e01078dca64" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:35.186975 32589 leader_election.cc:343] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.187095 32589 leader_election.cc:304] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.187302 979 ts_tablet_manager.cc:1918] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.187391 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet 36c8714cfbd84dd38cf5c0f305ef6843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:34 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:35.187528 979 log.cc:1199] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/f3d32798168345edb9898751d083057c
I20250623 08:04:35.187563 1128 raft_consensus.cc:2747] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.188422 979 ts_tablet_manager.cc:1939] T f3d32798168345edb9898751d083057c P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:35.189385 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 79) took 4024 ms. Trace:
I20250623 08:04:35.189535 342 ts_tablet_manager.cc:1397] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.290s user 0.038s sys 0.071s
I20250623 08:04:35.189469 979 rpcz_store.cc:276] 0623 08:04:31.164508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.164558 (+ 50us) service_pool.cc:225] Handling call
0623 08:04:35.135900 (+3971342us) ts_tablet_manager.cc:1145] Deleting tablet f3d32798168345edb9898751d083057c
0623 08:04:35.135910 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.166303 (+ 30393us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.187174 (+ 20871us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.189370 (+ 2196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3971006}
I20250623 08:04:35.189874 979 tablet_replica.cc:331] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:35.190052 979 raft_consensus.cc:2241] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:35.190224 979 raft_consensus.cc:2270] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.190738 979 ts_tablet_manager.cc:1905] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.194070 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef663ffdaf141aea5e6269b6b2e1c89" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:35.194619 32314 leader_election.cc:343] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 9bc95f12ab5d4b9b9f0729cb88aabbaa
W20250623 08:04:35.194696 32589 leader_election.cc:343] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.194824 32589 leader_election.cc:304] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.195019 1278 raft_consensus.cc:2747] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.195447 32314 leader_election.cc:304] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:35.196053 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a0b9402a264800a1ea39707c10da0c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.196246 32656 raft_consensus.cc:3058] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.199008 32656 raft_consensus.cc:2466] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.203590 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 67c44697ecab4dd9b43f544a602897ca (table default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]) successfully deleted
I20250623 08:04:35.208389 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:35.208822 32059 catalog_manager.cc:5582] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:35.209220 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:35.209590 1159 leader_election.cc:290] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.209595 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_32db9f68dd974fb8aff372a8eeccce3d [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 1 got 0
I20250623 08:04:35.209883 32059 catalog_manager.cc:5582] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:35.210225 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:35.210282 1303 raft_consensus.cc:2747] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.210952 952 ts_tablet_manager.cc:1918] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.212383 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc95f12ab5d4b9b9f0729cb88aabbaa" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.214459 848 raft_consensus.cc:513] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.215240 848 leader_election.cc:290] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.217160 342 raft_consensus.cc:357] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.217602 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet 36c8714cfbd84dd38cf5c0f305ef6843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:34 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.219030 32059 catalog_manager.cc:5582] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:35.220165 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:35.220664 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:35.221271 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
W20250623 08:04:35.221887 32589 leader_election.cc:343] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.222013 32589 leader_election.cc:304] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.222024 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 18ea63263b9b466e922169eecafab747 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]). Expected version 1 got 0
I20250623 08:04:35.222277 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet db81250d1e164a7d94affd764fc0ff0c (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:35.222395 1278 raft_consensus.cc:2747] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.222498 32059 catalog_manager.cc:5582] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:35.224663 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:35.225920 342 raft_consensus.cc:383] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.226025 342 raft_consensus.cc:738] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.226338 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:35.226636 342 consensus_queue.cc:260] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.227053 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet d5cd316e6f4447be958611ab07ed31fc (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:35.227278 342 ts_tablet_manager.cc:1428] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20250623 08:04:35.227835 342 tablet_bootstrap.cc:492] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:35.225706 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 4fe30079045d437aa61bb5bfbfb96133 because the tablet was not found. No further retry: Not found: Tablet not found: 4fe30079045d437aa61bb5bfbfb96133
I20250623 08:04:35.228178 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet f3d32798168345edb9898751d083057c (table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]) successfully deleted
I20250623 08:04:35.228832 952 log.cc:1199] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/8dc696236a15420e9666cca708068fb2
I20250623 08:04:35.228763 1332 raft_consensus.cc:513] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:35.229413 1332 leader_election.cc:290] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.229336 32059 catalog_manager.cc:5582] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:35.229718 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:35.229956 32059 catalog_manager.cc:5582] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:35.230340 342 tablet_bootstrap.cc:654] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.231938 952 ts_tablet_manager.cc:1939] T 8dc696236a15420e9666cca708068fb2 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:35.232373 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38ecd4c243b840f4b66d8e01078dca64" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.232018 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:35.232759 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:35.232937 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:35.233187 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_049497d4290c4bdfaf1e609b8b2ac9ac [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 2 got 0
I20250623 08:04:35.233474 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f83f68ebda7e4826bbb17278cd4ac5a8 (table default.Table_6acb69304c424830a635f22f12343b31 [id=c1782341d0d34662889cff00d557ff81]). Expected version 1 got 0
I20250623 08:04:35.233646 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:35.233896 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 136e482995a24c5daec55a701f3091fa (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:35.234211 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:35.234597 848 raft_consensus.cc:491] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.234783 848 raft_consensus.cc:513] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.235514 848 leader_election.cc:290] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.236629 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:35.238155 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_aaa3ca8d9bb0406981f0dfff8cbf9e3c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 4 got 0
I20250623 08:04:35.238989 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 771ecfc41e2b4007a4d213cec5b128c6 (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:35.239470 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:35.240450 32381 raft_consensus.cc:2391] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 2: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:35.240690 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:35.241146 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:35.241551 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b84b8b8713774ab4896842feb9fb075e (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:35.241925 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:35.242715 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 45) took 4269 ms. Trace:
I20250623 08:04:35.242841 952 rpcz_store.cc:276] 0623 08:04:30.962486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.962812 (+ 326us) service_pool.cc:225] Handling call
0623 08:04:35.151601 (+4188789us) ts_tablet_manager.cc:1145] Deleting tablet 8dc696236a15420e9666cca708068fb2
0623 08:04:35.151612 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.168196 (+ 16584us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.201177 (+ 32981us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.232244 (+ 31067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4188564}
I20250623 08:04:35.243257 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 46) took 4280 ms. Trace:
I20250623 08:04:35.243338 952 rpcz_store.cc:276] 0623 08:04:30.963157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.963376 (+ 219us) service_pool.cc:225] Handling call
0623 08:04:35.243094 (+4279718us) ts_tablet_manager.cc:1145] Deleting tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3
0623 08:04:35.243104 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.243211 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4279522}
I20250623 08:04:35.243662 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 47) took 4279 ms. Trace:
I20250623 08:04:35.243752 952 rpcz_store.cc:276] 0623 08:04:30.963730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.964071 (+ 341us) service_pool.cc:225] Handling call
0623 08:04:35.243553 (+4279482us) ts_tablet_manager.cc:1145] Deleting tablet 4fe30079045d437aa61bb5bfbfb96133
0623 08:04:35.243562 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.243617 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4279305}
I20250623 08:04:35.244025 952 tablet_replica.cc:331] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:35.244217 952 raft_consensus.cc:2241] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:35.244392 952 raft_consensus.cc:2270] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.244891 952 ts_tablet_manager.cc:1905] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.249599 32055 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 219) took 1466 ms. Trace:
I20250623 08:04:35.249722 32055 rpcz_store.cc:276] 0623 08:04:33.783495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.135307 (+1351812us) service_pool.cc:225] Handling call
0623 08:04:35.159226 (+ 23919us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:35.159748 (+ 522us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.249320 (+ 89572us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.249443 (+ 123us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.249561 (+ 118us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.162052 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.162183 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.162191 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:35.162194 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:35.162299 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.162312 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.162316 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.162438 (+ 122us) tablet.cc:699] Row locks acquired
0623 08:04:35.162441 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:35.162480 (+ 39us) write_op.cc:270] Start()
0623 08:04:35.162519 (+ 39us) write_op.cc:276] Timestamp: P: 1750665875162419 usec, L: 0
0623 08:04:35.162525 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.162714 (+ 189us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.247900 (+ 85186us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.248583 (+ 683us) write_op.cc:301] APPLY: starting
0623 08:04:35.248592 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.248626 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.248654 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.248764 (+ 110us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.248790 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.248796 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:35.248851 (+ 55us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.248892 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.248905 (+ 13us) write_op.cc:454] Released schema lock
0623 08:04:35.248910 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":85760,"child_traces":[["op",{"apply.queue_time_us":655,"apply.run_cpu_time_us":380,"apply.run_wall_time_us":380,"num_ops":2,"prepare.queue_time_us":1749,"prepare.run_cpu_time_us":760,"prepare.run_wall_time_us":760,"replication_time_us":85358,"threads_started":1}]]}
I20250623 08:04:35.258140 952 ts_tablet_manager.cc:1918] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:35.258277 952 log.cc:1199] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/162308da898d4dde8ab5b1ab01d0f393
I20250623 08:04:35.258672 952 ts_tablet_manager.cc:1939] T 162308da898d4dde8ab5b1ab01d0f393 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:35.259070 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_bc1251ce708a4467b5b1c71a1df307e0 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 1 got 0
I20250623 08:04:35.260039 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:35.261370 32050 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_3af33890e8ff40c881af74580254633b [id=8b20894a91c54a3eb08d5e1b4a42d269] alter complete (version 3)
W20250623 08:04:35.264570 32453 leader_election.cc:343] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 9bc95f12ab5d4b9b9f0729cb88aabbaa
I20250623 08:04:35.270851 32453 leader_election.cc:304] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:35.271127 1159 raft_consensus.cc:2802] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.271353 1159 raft_consensus.cc:695] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.271677 1159 consensus_queue.cc:237] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.273144 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdc9d47f2232403fa71e14a6fe6f33c3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.273334 32646 raft_consensus.cc:3058] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:35.274106 32051 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:39129 (request call id 171) took 1040 ms (client timeout 1000 ms). Trace:
W20250623 08:04:35.274209 32051 rpcz_store.cc:269] 0623 08:04:34.233590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.359918 (+126328us) service_pool.cc:225] Handling call
0623 08:04:35.274042 (+914124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57888,"reported_tablets":46}
I20250623 08:04:35.276911 32646 raft_consensus.cc:2466] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.278102 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f279b63b55c64e548ded27b02025c9f5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.278301 32646 raft_consensus.cc:2466] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:35.278690 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dc696236a15420e9666cca708068fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.278935 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet f3d32798168345edb9898751d083057c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.279153 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet d0ee5a4fe03848229524cc0105226d8c with delete_type TABLET_DATA_DELETED (replaced by tablet 38ecd4c243b840f4b66d8e01078dca64) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.279336 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a0b9402a264800a1ea39707c10da0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.281139 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 8dc696236a15420e9666cca708068fb2 (table default.Table_f81a1165815543b781251c1ecfd06779 [id=491b95490f8c45eda181ea0cafd42368]) successfully deleted
W20250623 08:04:35.282660 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 already in progress: creating tablet
W20250623 08:04:35.283210 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 4fe30079045d437aa61bb5bfbfb96133 because the tablet was not found. No further retry: Not found: Tablet not found: 4fe30079045d437aa61bb5bfbfb96133
I20250623 08:04:35.283701 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a0b9402a264800a1ea39707c10da0c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:35.283759 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 48) took 4319 ms. Trace:
I20250623 08:04:35.283885 952 rpcz_store.cc:276] 0623 08:04:30.964412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.964764 (+ 352us) service_pool.cc:225] Handling call
0623 08:04:35.243946 (+4279182us) ts_tablet_manager.cc:1145] Deleting tablet 162308da898d4dde8ab5b1ab01d0f393
0623 08:04:35.243954 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.249926 (+ 5972us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.258010 (+ 8084us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.283719 (+ 25709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4279021}
I20250623 08:04:35.284803 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdc9d47f2232403fa71e14a6fe6f33c3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:35.291738 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet c6a97b34e386443fa19a9601621b5017 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.291987 342 tablet_bootstrap.cc:492] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.292166 342 ts_tablet_manager.cc:1397] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.006s
I20250623 08:04:35.293206 342 raft_consensus.cc:357] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.293478 342 raft_consensus.cc:383] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.293769 342 raft_consensus.cc:738] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.294046 342 consensus_queue.cc:260] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.294688 342 ts_tablet_manager.cc:1428] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:35.295064 342 tablet_bootstrap.cc:492] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:35.295969 952 tablet_replica.cc:331] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:35.296272 952 raft_consensus.cc:2241] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.296519 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.296875 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ecf4af9e476495683bd63cac20d4672 with delete_type TABLET_DATA_DELETED (replaced by tablet 52d0a356c74f4daea9093c998b4bf73e) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.297461 952 pending_rounds.cc:70] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:35.298120 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f279b63b55c64e548ded27b02025c9f5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.299814 952 pending_rounds.cc:77] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727421274841088 op_type: NO_OP noop_request { }
I20250623 08:04:35.299970 952 raft_consensus.cc:2887] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:35.300117 952 raft_consensus.cc:2270] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.300529 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.300976 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 279624b8bb834b95a4f08045968ebfb4 with delete_type TABLET_DATA_DELETED (replaced by tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.301791 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.302605 952 ts_tablet_manager.cc:1905] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.303082 981 ts_tablet_manager.cc:1918] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.303253 981 log.cc:1199] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/228375592aee48a1aa21e1d16962b663
I20250623 08:04:35.299553 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet 2060a643d0da47efa868f52bdb331b1f with delete_type TABLET_DATA_DELETED (replaced by tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.302838 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet a414144276f6484ea4690204752e2142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.303144 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.304039 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.304350 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 2795cb14e6db4116a4c9ee539bd4e82d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:35.308041 32050 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee], current schema version=5): Not found: failed to reset TS proxy: No leader found: Tablet a8de2b1105564fdab386442b1de4cedb consensus state has no leader
I20250623 08:04:35.309887 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc95f12ab5d4b9b9f0729cb88aabbaa" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:35.310129 32513 raft_consensus.cc:3058] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:35.313208 32513 raft_consensus.cc:2466] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 2.
I20250623 08:04:35.313877 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 with delete_type TABLET_DATA_DELETED (replaced by tablet 306c4021b5d94a4996de07c222a8bfd8) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.299238 342 tablet_bootstrap.cc:654] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.314375 343 tablet_bootstrap.cc:492] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
W20250623 08:04:35.315433 32050 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:39501 (request call id 167) took 1648 ms (client timeout 1000 ms). Trace:
I20250623 08:04:35.315495 343 ts_tablet_manager.cc:1397] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.147s user 0.003s sys 0.001s
I20250623 08:04:35.316330 343 raft_consensus.cc:357] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.316516 343 raft_consensus.cc:383] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 08:04:35.315899 32057 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet bb06a340f82047ac8ec3516edb08f4b6 consensus state has no leader
I20250623 08:04:35.316568 343 raft_consensus.cc:738] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.316787 343 consensus_queue.cc:260] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.317307 343 ts_tablet_manager.cc:1428] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:35.318051 32314 leader_election.cc:304] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:35.318392 32703 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:35.319027 1332 raft_consensus.cc:2747] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.320294 981 ts_tablet_manager.cc:1939] T 228375592aee48a1aa21e1d16962b663 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:35.321844 32314 leader_election.cc:304] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:35.322533 1332 raft_consensus.cc:2802] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election won for term 2
W20250623 08:04:35.316257 32050 rpcz_store.cc:269] 0623 08:04:33.666980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.734453 (+ 67473us) service_pool.cc:225] Handling call
0623 08:04:35.315360 (+1580907us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:34.365229 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.365371 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.365378 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:34.365381 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:34.365500 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.365515 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.365520 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:34.365591 (+ 71us) tablet.cc:699] Row locks acquired
0623 08:04:34.365594 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:34.365639 (+ 45us) write_op.cc:270] Start()
0623 08:04:34.365684 (+ 45us) write_op.cc:276] Timestamp: P: 1750665874365607 usec, L: 0
0623 08:04:34.365689 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.365937 (+ 248us) log.cc:844] Serialized 527 byte log entry
0623 08:04:34.379715 (+ 13778us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.379813 (+ 98us) write_op.cc:301] APPLY: starting
0623 08:04:34.379832 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.379878 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.379913 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.379973 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.380008 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:34.380014 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:34.380093 (+ 79us) log.cc:844] Serialized 23 byte log entry
0623 08:04:34.380166 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.380187 (+ 21us) write_op.cc:454] Released schema lock
0623 08:04:34.380196 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:34.380269 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.380273 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:34.401217 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.401370 (+ 153us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.401377 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:34.401379 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:34.401509 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.401523 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.401531 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:34.401609 (+ 78us) tablet.cc:699] Row locks acquired
0623 08:04:34.401614 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:34.401660 (+ 46us) write_op.cc:270] Start()
0623 08:04:34.401694 (+ 34us) write_op.cc:276] Timestamp: P: 1750665874401627 usec, L: 0
0623 08:04:34.401699 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.401870 (+ 171us) log.cc:844] Serialized 356 byte log entry
0623 08:04:34.409885 (+ 8015us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.410045 (+ 160us) write_op.cc:301] APPLY: starting
0623 08:04:34.410063 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.410133 (+ 70us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.410173 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.410233 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.410277 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:34.410288 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:34.410372 (+ 84us) log.cc:844] Serialized 23 byte log entry
0623 08:04:34.410437 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.410460 (+ 23us) write_op.cc:454] Released schema lock
0623 08:04:34.410469 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:34.410545 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.410548 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:34.423295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:34.423379 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:34.423383 (+ 4us) write_op.cc:435] Acquired schema lock
0623 08:04:34.423386 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:34.423459 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:34.423469 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:34.423473 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:34.423521 (+ 48us) tablet.cc:699] Row locks acquired
0623 08:04:34.423522 (+ 1us) write_op.cc:260] PREPARE: finished
0623 08:04:34.423548 (+ 26us) write_op.cc:270] Start()
0623 08:04:34.430738 (+ 7190us) write_op.cc:276] Timestamp: P: 1750665874423517 usec, L: 0
0623 08:04:34.430743 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:34.430957 (+ 214us) log.cc:844] Serialized 356 byte log entry
0623 08:04:34.476749 (+ 45792us) op_driver.cc:464] REPLICATION: finished
0623 08:04:34.477582 (+ 833us) write_op.cc:301] APPLY: starting
0623 08:04:34.477595 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:34.477626 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:34.477662 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:34.477706 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:34.477741 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:34.477751 (+ 10us) write_op.cc:312] APPLY: finished
0623 08:04:34.477806 (+ 55us) log.cc:844] Serialized 23 byte log entry
0623 08:04:34.477849 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:34.477863 (+ 14us) write_op.cc:454] Released schema lock
0623 08:04:34.477870 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0623 08:04:35.129857 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.129967 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.129973 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:35.129975 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:35.130082 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.130098 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.130105 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:35.130177 (+ 72us) tablet.cc:699] Row locks acquired
0623 08:04:35.130182 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:35.130222 (+ 40us) write_op.cc:270] Start()
0623 08:04:35.130260 (+ 38us) write_op.cc:276] Timestamp: P: 1750665875130167 usec, L: 0
0623 08:04:35.130266 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.130522 (+ 256us) log.cc:844] Serialized 356 byte log entry
0623 08:04:35.247830 (+117308us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.248072 (+ 242us) write_op.cc:301] APPLY: starting
0623 08:04:35.248090 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.248132 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.248176 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.248231 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.248267 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:35.248274 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.248361 (+ 87us) log.cc:844] Serialized 23 byte log entry
0623 08:04:35.248424 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.248444 (+ 20us) write_op.cc:454] Released schema lock
0623 08:04:35.248451 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.248518 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.248522 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":15391,"reported_tablets":150,"child_traces":[["op",{"apply.queue_time_us":33,"apply.run_cpu_time_us":451,"apply.run_wall_time_us":450,"num_ops":1,"prepare.queue_time_us":10240,"prepare.run_cpu_time_us":832,"prepare.run_wall_time_us":831,"raft.queue_time_us":1062,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":313,"replication_time_us":13988}],["op",{"apply.queue_time_us":107,"apply.run_cpu_time_us":493,"apply.run_wall_time_us":493,"num_ops":1,"prepare.queue_time_us":1234,"prepare.run_cpu_time_us":754,"prepare.run_wall_time_us":754,"raft.queue_time_us":444,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":9,"replication_time_us":8145}],["op",{"apply.queue_time_us":794,"apply.run_cpu_time_us":349,"apply.run_wall_time_us":350,"num_ops":1,"prepare.queue_time_us":6353,"prepare.run_cpu_time_us":742,"prepare.run_wall_time_us":7761,"raft.queue_time_us":67,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":45968,"thread_start_us":21224,"threads_started":1}],["op",{"apply.queue_time_us":187,"apply.run_cpu_time_us":440,"apply.run_wall_time_us":439,"num_ops":1,"prepare.queue_time_us":2815,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":746,"replication_time_us":117530}]]}
I20250623 08:04:35.322829 1332 raft_consensus.cc:695] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:35.323153 1332 consensus_queue.cc:237] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:35.323411 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a0b9402a264800a1ea39707c10da0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.323797 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 726d80e05a344f27b5a289723cb5aed7 with delete_type TABLET_DATA_DELETED (replaced by tablet 432d1642c0e44119b5bc05858463a136) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.324503 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet b980c746a3c042bfae90aa076acbb1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.324891 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 32c82141edea4068b54755a85c6d7fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 162308da898d4dde8ab5b1ab01d0f393) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.325296 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet d4221d7500d1458db75926d7b274a654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.325665 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 0311ce67989f49e8866234a88e5b4d09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.326011 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 67c44697ecab4dd9b43f544a602897ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.326402 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.326789 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dc696236a15420e9666cca708068fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.327862 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 162308da898d4dde8ab5b1ab01d0f393 (table default.Table_c30b48097bd643b18b2a744839780b9e [id=7e59ad2235c1418d97f2b73ff2a48300]) successfully deleted
W20250623 08:04:35.328111 32453 consensus_peers.cc:487] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:35.326839 32057 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.1:45613 (request call id 154) took 1331 ms (client timeout 1000 ms). Trace:
W20250623 08:04:35.328598 32057 rpcz_store.cc:269] 0623 08:04:33.995169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.354901 (+359732us) service_pool.cc:225] Handling call
0623 08:04:35.326775 (+971874us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":160}
I20250623 08:04:35.329252 32453 leader_election.cc:304] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.329526 848 raft_consensus.cc:2747] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 08:04:35.329830 32453 leader_election.cc:343] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.330125 32453 consensus_peers.cc:487] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:35.330303 32453 leader_election.cc:343] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.330647 32453 leader_election.cc:343] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.330937 32453 leader_election.cc:304] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.331197 1366 raft_consensus.cc:2802] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.331331 1366 raft_consensus.cc:695] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.331597 1366 consensus_queue.cc:237] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.339282 1278 raft_consensus.cc:491] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.339432 1278 raft_consensus.cc:513] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.339994 1278 leader_election.cc:290] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:35.340726 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c58231f67604422a1497e4c1b44ec52" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:35.341315 32589 leader_election.cc:343] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.343163 1303 raft_consensus.cc:491] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.343320 1303 raft_consensus.cc:513] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.343981 1303 leader_election.cc:290] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.346329 343 tablet_bootstrap.cc:492] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.348754 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 80) took 2934 ms. Trace:
I20250623 08:04:35.351718 342 tablet_bootstrap.cc:492] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.351898 342 ts_tablet_manager.cc:1397] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.002s
I20250623 08:04:35.352680 342 raft_consensus.cc:357] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.352888 342 raft_consensus.cc:383] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.352974 342 raft_consensus.cc:738] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.353247 342 consensus_queue.cc:260] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.353821 342 ts_tablet_manager.cc:1428] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:35.354131 342 tablet_bootstrap.cc:492] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:35.357470 952 ts_tablet_manager.cc:1918] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.361625 981 rpcz_store.cc:276] 0623 08:04:32.414521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.415088 (+ 567us) service_pool.cc:225] Handling call
0623 08:04:34.772643 (+2357555us) ts_tablet_manager.cc:1145] Deleting tablet 228375592aee48a1aa21e1d16962b663
0623 08:04:34.772653 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.089339 (+316686us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.302917 (+213578us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.348681 (+ 45764us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27445,"tablet-delete.queue_time_us":2357301}
I20250623 08:04:35.362143 981 tablet_replica.cc:331] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:35.362471 981 raft_consensus.cc:2241] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.362946 981 raft_consensus.cc:2270] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.363668 981 ts_tablet_manager.cc:1905] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.366925 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:35.368448 343 tablet_bootstrap.cc:654] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.373775 952 log.cc:1199] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/205da880685b49b3b8669f73dc2c563b
I20250623 08:04:35.331310 32453 leader_election.cc:304] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.358794 342 tablet_bootstrap.cc:654] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.375101 1366 raft_consensus.cc:2802] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.375224 1366 raft_consensus.cc:491] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.375329 1366 raft_consensus.cc:3058] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.374318 952 ts_tablet_manager.cc:1939] T 205da880685b49b3b8669f73dc2c563b P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:35.376724 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 49) took 4411 ms. Trace:
I20250623 08:04:35.355087 339 tablet_bootstrap.cc:492] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:35.376899 952 rpcz_store.cc:276] 0623 08:04:30.965062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.965347 (+ 285us) service_pool.cc:225] Handling call
0623 08:04:35.291472 (+4326125us) ts_tablet_manager.cc:1145] Deleting tablet 205da880685b49b3b8669f73dc2c563b
0623 08:04:35.291481 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.308423 (+ 16942us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.357312 (+ 48889us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.376675 (+ 19363us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4325903}
I20250623 08:04:35.377295 952 tablet_replica.cc:331] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:35.365542 32568 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:35.355531 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 32c82141edea4068b54755a85c6d7fb8 with delete_type TABLET_DATA_DELETED (replaced by tablet 162308da898d4dde8ab5b1ab01d0f393) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:35.355849 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet b980c746a3c042bfae90aa076acbb1df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:35.378137 1366 raft_consensus.cc:513] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.378221 32052 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 217) took 1662 ms. Trace:
I20250623 08:04:35.378334 32052 rpcz_store.cc:276] 0623 08:04:33.715648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.124888 (+1409240us) service_pool.cc:225] Handling call
0623 08:04:35.159135 (+ 34247us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:35.159754 (+ 619us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.377929 (+218175us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.378057 (+ 128us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.378154 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.162845 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.162945 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.162950 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:35.162953 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:35.163032 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.163041 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.163044 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.163103 (+ 59us) tablet.cc:699] Row locks acquired
0623 08:04:35.163106 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:35.163133 (+ 27us) write_op.cc:270] Start()
0623 08:04:35.163159 (+ 26us) write_op.cc:276] Timestamp: P: 1750665875163076 usec, L: 0
0623 08:04:35.163162 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.163296 (+ 134us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.377161 (+213865us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.377329 (+ 168us) write_op.cc:301] APPLY: starting
0623 08:04:35.377347 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.377394 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.377421 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.377600 (+ 179us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.377660 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.377667 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.377752 (+ 85us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.377817 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.377834 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:35.377841 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.377941 (+ 100us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.377945 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":18304,"child_traces":[["op",{"apply.queue_time_us":111,"apply.run_cpu_time_us":585,"apply.run_wall_time_us":586,"mutex_wait_us":1686,"num_ops":2,"prepare.queue_time_us":2562,"prepare.run_cpu_time_us":529,"prepare.run_wall_time_us":529,"replication_time_us":213909}]]}
I20250623 08:04:35.379124 1366 leader_election.cc:290] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.379369 32056 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 221) took 1438 ms. Trace:
I20250623 08:04:35.379499 32056 rpcz_store.cc:276] 0623 08:04:33.941231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.137994 (+1196763us) service_pool.cc:225] Handling call
0623 08:04:35.160293 (+ 22299us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:35.160750 (+ 457us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.379139 (+218389us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.379238 (+ 99us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.379324 (+ 86us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.163407 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.163499 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.163504 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:35.163506 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:35.163571 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.163578 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.163582 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.163644 (+ 62us) tablet.cc:699] Row locks acquired
0623 08:04:35.163646 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:35.163672 (+ 26us) write_op.cc:270] Start()
0623 08:04:35.163694 (+ 22us) write_op.cc:276] Timestamp: P: 1750665875163616 usec, L: 0
0623 08:04:35.163697 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.163813 (+ 116us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.377245 (+213432us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.378011 (+ 766us) write_op.cc:301] APPLY: starting
0623 08:04:35.378026 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.378056 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.378083 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.378162 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.378196 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.378209 (+ 13us) write_op.cc:312] APPLY: finished
0623 08:04:35.378267 (+ 58us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.378310 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.378329 (+ 19us) write_op.cc:454] Released schema lock
0623 08:04:35.378336 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":733,"apply.run_cpu_time_us":386,"apply.run_wall_time_us":385,"mutex_wait_us":879,"num_ops":2,"prepare.queue_time_us":2246,"prepare.run_cpu_time_us":470,"prepare.run_wall_time_us":470,"replication_time_us":213531}]]}
W20250623 08:04:35.380239 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 48631us
I20250623 08:04:35.382148 32058 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 222) took 1440 ms. Trace:
I20250623 08:04:35.382242 32058 rpcz_store.cc:276] 0623 08:04:33.941330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.138359 (+1197029us) service_pool.cc:225] Handling call
0623 08:04:35.160638 (+ 22279us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:35.161015 (+ 377us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.381929 (+220914us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.382013 (+ 84us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.382083 (+ 70us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.163971 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.164051 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.164055 (+ 4us) write_op.cc:435] Acquired schema lock
0623 08:04:35.164057 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:35.164137 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.164144 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.164148 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.164232 (+ 84us) tablet.cc:699] Row locks acquired
0623 08:04:35.164234 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:35.164260 (+ 26us) write_op.cc:270] Start()
0623 08:04:35.164285 (+ 25us) write_op.cc:276] Timestamp: P: 1750665875164203 usec, L: 0
0623 08:04:35.164290 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.164425 (+ 135us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.381206 (+216781us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.381518 (+ 312us) write_op.cc:301] APPLY: starting
0623 08:04:35.381531 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.381562 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.381591 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.381689 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.381720 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.381727 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.381780 (+ 53us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.381831 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.381848 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:35.381858 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":37,"apply.run_cpu_time_us":406,"apply.run_wall_time_us":406,"mutex_wait_us":2497,"num_ops":2,"prepare.queue_time_us":2543,"prepare.run_cpu_time_us":514,"prepare.run_wall_time_us":514,"replication_time_us":216890}]]}
I20250623 08:04:35.382894 32058 master_service.cc:948] ReplaceTablet: received request to replace tablet 789d1c20aa72480ab71123c799bbde4c from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:35.383224 32053 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 218) took 1601 ms. Trace:
I20250623 08:04:35.383494 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 205da880685b49b3b8669f73dc2c563b (table default.Table_4d4535cab44545839eb3a559385e395a [id=bb849e86d7084959830c68207fb99061]) successfully deleted
I20250623 08:04:35.384421 339 ts_tablet_manager.cc:1397] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.341s user 0.005s sys 0.011s
I20250623 08:04:35.383343 32053 rpcz_store.cc:276] 0623 08:04:33.782062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.131879 (+1349817us) service_pool.cc:225] Handling call
0623 08:04:35.161157 (+ 29278us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:35.161566 (+ 409us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.383014 (+221448us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.383097 (+ 83us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.383186 (+ 89us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.164529 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.164619 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.164623 (+ 4us) write_op.cc:435] Acquired schema lock
0623 08:04:35.164626 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:35.164699 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.164706 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.164710 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.164764 (+ 54us) tablet.cc:699] Row locks acquired
0623 08:04:35.164768 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:35.164794 (+ 26us) write_op.cc:270] Start()
0623 08:04:35.164819 (+ 25us) write_op.cc:276] Timestamp: P: 1750665875164739 usec, L: 0
0623 08:04:35.164823 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.164944 (+ 121us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.382300 (+217356us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.382397 (+ 97us) write_op.cc:301] APPLY: starting
0623 08:04:35.382409 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.382456 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.382477 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.382559 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.382590 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.382597 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.382659 (+ 62us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.382711 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.382730 (+ 19us) write_op.cc:454] Released schema lock
0623 08:04:35.382737 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":417,"apply.run_wall_time_us":418,"mutex_wait_us":2254,"num_ops":2,"prepare.queue_time_us":2594,"prepare.run_cpu_time_us":523,"prepare.run_wall_time_us":522,"replication_time_us":217454}]]}
I20250623 08:04:35.384667 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f279b63b55c64e548ded27b02025c9f5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.384852 32646 raft_consensus.cc:3058] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.385654 339 raft_consensus.cc:357] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.385854 339 raft_consensus.cc:383] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.385929 339 raft_consensus.cc:738] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.386647 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_59f2123c149e43569f3e3db991d1d09c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:35.386862 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 228375592aee48a1aa21e1d16962b663 (table default.Table_10f030b47f8e4b31ba84115caef0203d [id=1cac605ff7f8488dbb5c2f95fc31dec2]) successfully deleted
W20250623 08:04:35.387291 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59f2123c149e43569f3e3db991d1d09c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:35.388047 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c58231f67604422a1497e4c1b44ec52" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.388471 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_d8bf0b849fe24798821c0c3ebe8745ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:35.388944 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8bf0b849fe24798821c0c3ebe8745ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:35.389778 32589 leader_election.cc:343] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.389900 32589 leader_election.cc:304] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.390245 1278 raft_consensus.cc:2747] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.391083 32646 raft_consensus.cc:2466] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.392338 1366 consensus_queue.cc:1035] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:35.392585 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_8a32f131cc9048a583663a64ed26a4af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:35.393127 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a32f131cc9048a583663a64ed26a4af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:35.393383 32453 leader_election.cc:304] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:35.394270 1366 raft_consensus.cc:2802] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.394784 1366 raft_consensus.cc:695] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.395325 1366 consensus_queue.cc:237] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.378255 952 raft_consensus.cc:2241] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:35.396469 952 raft_consensus.cc:2270] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.397059 952 ts_tablet_manager.cc:1905] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.400300 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_31ab57df769c4b2cb4e2cb74d69b4eb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:35.400848 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31ab57df769c4b2cb4e2cb74d69b4eb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:35.402391 32053 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_2c99afea2b6e4963985dab32b545779e" } new_table_name: "default.Table_d281efc20a4542faa2838570433fd540" modify_external_catalogs: true
I20250623 08:04:35.403079 342 tablet_bootstrap.cc:492] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.403175 1278 raft_consensus.cc:491] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.403221 342 ts_tablet_manager.cc:1397] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.004s
I20250623 08:04:35.403297 1278 raft_consensus.cc:513] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.403873 1278 leader_election.cc:290] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:35.403975 342 raft_consensus.cc:357] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.404137 342 raft_consensus.cc:383] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.404232 342 raft_consensus.cc:738] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.404460 342 consensus_queue.cc:260] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.404994 342 ts_tablet_manager.cc:1428] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:35.405264 342 tablet_bootstrap.cc:492] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:35.405318 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d98654c29b9c4e75af6a213cafd4c2c9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:35.405891 32589 leader_election.cc:343] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.409073 32314 leader_election.cc:343] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 9bc95f12ab5d4b9b9f0729cb88aabbaa
I20250623 08:04:35.409360 339 consensus_queue.cc:260] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.411350 339 ts_tablet_manager.cc:1428] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250623 08:04:35.411803 342 tablet_bootstrap.cc:654] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.412528 32051 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 224) took 1438 ms. Trace:
I20250623 08:04:35.412648 32051 rpcz_store.cc:276] 0623 08:04:33.973788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.276931 (+1303143us) service_pool.cc:225] Handling call
0623 08:04:35.388407 (+111476us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f9313 00007f19c7642c96 00007f19c76429c5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:35.389056 (+ 649us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:35.389464 (+ 408us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.412257 (+ 22793us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.412373 (+ 116us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.412469 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.398065 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.398146 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.398152 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:35.398154 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:35.398222 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.398230 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.398234 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.398292 (+ 58us) tablet.cc:699] Row locks acquired
0623 08:04:35.398294 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:35.398322 (+ 28us) write_op.cc:270] Start()
0623 08:04:35.398347 (+ 25us) write_op.cc:276] Timestamp: P: 1750665875398293 usec, L: 0
0623 08:04:35.398351 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.398475 (+ 124us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.406704 (+ 8229us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.407439 (+ 735us) write_op.cc:301] APPLY: starting
0623 08:04:35.407453 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.407481 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.407505 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.407574 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.407598 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.407605 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.407658 (+ 53us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.407692 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.407702 (+ 10us) write_op.cc:454] Released schema lock
0623 08:04:35.407707 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":255729024,"child_traces":[["op",{"apply.queue_time_us":703,"apply.run_cpu_time_us":330,"apply.run_wall_time_us":330,"num_ops":2,"prepare.queue_time_us":8167,"prepare.run_cpu_time_us":467,"prepare.run_wall_time_us":465,"replication_time_us":8338,"threads_started":1}]]}
I20250623 08:04:35.413420 343 tablet_bootstrap.cc:492] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:35.413537 343 ts_tablet_manager.cc:1397] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.005s
I20250623 08:04:35.414269 343 raft_consensus.cc:357] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.414433 343 raft_consensus.cc:383] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.414521 343 raft_consensus.cc:738] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.414770 343 consensus_queue.cc:260] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.414922 32057 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 225) took 1440 ms. Trace:
I20250623 08:04:35.415021 32057 rpcz_store.cc:276] 0623 08:04:33.974116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.330684 (+1356568us) service_pool.cc:225] Handling call
0623 08:04:35.386545 (+ 55861us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f9313 00007f19c7642c96 00007f19c76429c5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:35.387376 (+ 831us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:35.387817 (+ 441us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.414688 (+ 26871us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.414794 (+ 106us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.414872 (+ 78us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.397381 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.397489 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.397501 (+ 12us) write_op.cc:435] Acquired schema lock
0623 08:04:35.397504 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:35.397607 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.397628 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.397635 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.397722 (+ 87us) tablet.cc:699] Row locks acquired
0623 08:04:35.397725 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:35.397758 (+ 33us) write_op.cc:270] Start()
0623 08:04:35.397790 (+ 32us) write_op.cc:276] Timestamp: P: 1750665875397727 usec, L: 0
0623 08:04:35.397795 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.397961 (+ 166us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.406647 (+ 8686us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.407257 (+ 610us) write_op.cc:301] APPLY: starting
0623 08:04:35.407268 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.407299 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.407326 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.407455 (+ 129us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.407485 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.407493 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:35.407546 (+ 53us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.407587 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.407599 (+ 12us) write_op.cc:454] Released schema lock
0623 08:04:35.407604 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.407665 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.407669 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":127704448,"child_traces":[["op",{"apply.queue_time_us":539,"apply.run_cpu_time_us":404,"apply.run_wall_time_us":405,"num_ops":2,"prepare.queue_time_us":9127,"prepare.run_cpu_time_us":653,"prepare.run_wall_time_us":654,"replication_time_us":8829}]]}
I20250623 08:04:35.415727 32057 master_service.cc:948] ReplaceTablet: received request to replace tablet db81250d1e164a7d94affd764fc0ff0c from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:35.417022 339 tablet_bootstrap.cc:492] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:35.411664 32314 leader_election.cc:343] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 4cb66c7a273a412db3ae77af9e30d036
I20250623 08:04:35.420936 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f279b63b55c64e548ded27b02025c9f5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:35.421516 343 ts_tablet_manager.cc:1428] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250623 08:04:35.422199 32314 leader_election.cc:343] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 4cb66c7a273a412db3ae77af9e30d036
I20250623 08:04:35.422334 32314 leader_election.cc:304] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:35.422678 1332 raft_consensus.cc:2747] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.429860 339 tablet_bootstrap.cc:654] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.432118 342 tablet_bootstrap.cc:492] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.432284 342 ts_tablet_manager.cc:1397] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250623 08:04:35.433095 342 raft_consensus.cc:357] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.433351 342 raft_consensus.cc:383] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.433458 342 raft_consensus.cc:738] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.433729 342 consensus_queue.cc:260] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.434338 342 ts_tablet_manager.cc:1428] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:35.434700 342 tablet_bootstrap.cc:492] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:35.446682 342 tablet_bootstrap.cc:654] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.437134 952 ts_tablet_manager.cc:1918] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250623 08:04:35.447944 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d98654c29b9c4e75af6a213cafd4c2c9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.453387 952 log.cc:1199] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/8d8f38a6de1e45e7bed4abdced684c92
I20250623 08:04:35.453872 952 ts_tablet_manager.cc:1939] T 8d8f38a6de1e45e7bed4abdced684c92 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:35.454262 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 50) took 4488 ms. Trace:
I20250623 08:04:35.454435 952 rpcz_store.cc:276] 0623 08:04:30.965735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.966098 (+ 363us) service_pool.cc:225] Handling call
0623 08:04:35.377230 (+4411132us) ts_tablet_manager.cc:1145] Deleting tablet 8d8f38a6de1e45e7bed4abdced684c92
0623 08:04:35.377239 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.407665 (+ 30426us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.436974 (+ 29309us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.454214 (+ 17240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4410962}
I20250623 08:04:35.454862 952 tablet_replica.cc:331] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:35.455108 952 raft_consensus.cc:2241] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.455518 952 raft_consensus.cc:2270] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.456115 952 ts_tablet_manager.cc:1905] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.462296 343 tablet_bootstrap.cc:492] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.464124 343 tablet_bootstrap.cc:654] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:35.466775 32589 leader_election.cc:343] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.466892 32589 leader_election.cc:304] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.467113 1278 raft_consensus.cc:2747] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.467612 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 8d8f38a6de1e45e7bed4abdced684c92 (table default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e [id=7a569a9fe3bd4c608ddca48a734df1da]) successfully deleted
I20250623 08:04:35.471086 32051 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_61ccd9db16cb4618874bff73413fd485" } modify_external_catalogs: true
I20250623 08:04:35.472129 979 ts_tablet_manager.cc:1918] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:35.472283 979 log.cc:1199] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/5fdc9e53da264a5ca1f70e1846c906f0
I20250623 08:04:35.472702 979 ts_tablet_manager.cc:1939] T 5fdc9e53da264a5ca1f70e1846c906f0 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:35.472724 32051 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_61ccd9db16cb4618874bff73413fd485" } modify_external_catalogs: true
I20250623 08:04:35.473902 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 80) took 4179 ms. Trace:
I20250623 08:04:35.473984 979 rpcz_store.cc:276] 0623 08:04:31.294702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.294940 (+ 238us) service_pool.cc:225] Handling call
0623 08:04:35.189833 (+3894893us) ts_tablet_manager.cc:1145] Deleting tablet 5fdc9e53da264a5ca1f70e1846c906f0
0623 08:04:35.189841 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.193671 (+ 3830us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.472021 (+278350us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.473886 (+ 1865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3894685}
I20250623 08:04:35.475891 979 tablet_replica.cc:331] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:35.476082 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 5fdc9e53da264a5ca1f70e1846c906f0 (table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]) successfully deleted
I20250623 08:04:35.477057 342 tablet_bootstrap.cc:492] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.477227 342 ts_tablet_manager.cc:1397] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.001s
I20250623 08:04:35.478192 342 raft_consensus.cc:357] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.478420 342 raft_consensus.cc:383] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.478523 342 raft_consensus.cc:738] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
W20250623 08:04:35.479207 32044 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet a8de2b1105564fdab386442b1de4cedb consensus state has no leader
W20250623 08:04:35.469586 32453 leader_election.cc:343] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.483400 32453 consensus_peers.cc:487] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:35.485774 32055 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 223) took 1544 ms. Trace:
I20250623 08:04:35.485949 32055 rpcz_store.cc:276] 0623 08:04:33.941422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.250267 (+1308845us) service_pool.cc:225] Handling call
0623 08:04:35.392543 (+142276us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f9313 00007f19c7642c96 00007f19c76429c5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:35.393217 (+ 674us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:35.393634 (+ 417us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.485459 (+ 91825us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.485564 (+ 105us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.485719 (+ 155us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.398557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.398653 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.398658 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:35.398660 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:35.398725 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.398732 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.398736 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.398820 (+ 84us) tablet.cc:699] Row locks acquired
0623 08:04:35.398822 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:35.398848 (+ 26us) write_op.cc:270] Start()
0623 08:04:35.398870 (+ 22us) write_op.cc:276] Timestamp: P: 1750665875398820 usec, L: 0
0623 08:04:35.398873 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.399007 (+ 134us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.469249 (+ 70242us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.484430 (+ 15181us) write_op.cc:301] APPLY: starting
0623 08:04:35.484445 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.484483 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.484506 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.484643 (+ 137us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.484677 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.484683 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:35.484752 (+ 69us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.484831 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.484854 (+ 23us) write_op.cc:454] Released schema lock
0623 08:04:35.484862 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.484942 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.484946 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":326571264,"child_traces":[["op",{"apply.queue_time_us":15122,"apply.run_cpu_time_us":507,"apply.run_wall_time_us":506,"num_ops":2,"prepare.queue_time_us":4531,"prepare.run_cpu_time_us":496,"prepare.run_wall_time_us":495,"replication_time_us":70344}]]}
I20250623 08:04:35.487686 32056 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 226) took 1493 ms. Trace:
I20250623 08:04:35.487838 32056 rpcz_store.cc:276] 0623 08:04:33.994191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.380076 (+1385885us) service_pool.cc:225] Handling call
0623 08:04:35.400237 (+ 20161us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f9313 00007f19c7642c96 00007f19c76429c5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:35.400944 (+ 707us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:35.401355 (+ 411us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:35.487380 (+ 86025us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:35.487509 (+ 129us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:35.487632 (+ 123us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.404531 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.404609 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.404613 (+ 4us) write_op.cc:435] Acquired schema lock
0623 08:04:35.404625 (+ 12us) tablet.cc:660] Decoding operations
0623 08:04:35.404696 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.404703 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.404707 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.404763 (+ 56us) tablet.cc:699] Row locks acquired
0623 08:04:35.404766 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:35.404792 (+ 26us) write_op.cc:270] Start()
0623 08:04:35.404816 (+ 24us) write_op.cc:276] Timestamp: P: 1750665875404763 usec, L: 0
0623 08:04:35.404819 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.404960 (+ 141us) log.cc:844] Serialized 433 byte log entry
0623 08:04:35.484197 (+ 79237us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.485017 (+ 820us) write_op.cc:301] APPLY: starting
0623 08:04:35.485026 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.485079 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.485113 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.485195 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.485224 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:35.485231 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.485288 (+ 57us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.485328 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.485341 (+ 13us) write_op.cc:454] Released schema lock
0623 08:04:35.485346 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":45695872,"child_traces":[["op",{"apply.queue_time_us":778,"apply.run_cpu_time_us":381,"apply.run_wall_time_us":381,"num_ops":2,"prepare.queue_time_us":2756,"prepare.run_cpu_time_us":494,"prepare.run_wall_time_us":495,"replication_time_us":79350}]]}
I20250623 08:04:35.502168 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 501b5233522f43c7aed3c9770adb5c03
I20250623 08:04:35.502760 32054 catalog_manager.cc:6652] ReplaceTablet: tablet 501b5233522f43c7aed3c9770adb5c03 deleted and replaced by tablet 74c1cdc83ffd4a09a163479ac58fd405
I20250623 08:04:35.502882 32054 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36116 (request call id 220) took 1702 ms. Trace:
I20250623 08:04:35.502990 32054 rpcz_store.cc:276] 0623 08:04:33.800438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.136941 (+1336503us) service_pool.cc:225] Handling call
0623 08:04:35.502844 (+365903us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.434152 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.434316 (+ 164us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.434325 (+ 9us) write_op.cc:435] Acquired schema lock
0623 08:04:35.434330 (+ 5us) tablet.cc:660] Decoding operations
0623 08:04:35.434467 (+ 137us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.434481 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.434489 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.434680 (+ 191us) tablet.cc:699] Row locks acquired
0623 08:04:35.434685 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:35.434735 (+ 50us) write_op.cc:270] Start()
0623 08:04:35.434790 (+ 55us) write_op.cc:276] Timestamp: P: 1750665875434695 usec, L: 0
0623 08:04:35.434797 (+ 7us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.469486 (+ 34689us) log.cc:844] Serialized 622 byte log entry
0623 08:04:35.469680 (+ 194us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 000000000097041a 00007f19c8ac25e2 00007f19c9e69255 00007f19c9e67d79 00007f19c9e701a0 00007f19c9e6ffad 0000000000a751bc 00007f19bd0d43ff 00007f19bd0d76cb 00007f19bd0d7548 0000000000a751bc
0623 08:04:35.501407 (+ 31727us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.501474 (+ 67us) write_op.cc:301] APPLY: starting
0623 08:04:35.501483 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.501511 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.501513 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.501587 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.501630 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:35.501640 (+ 10us) write_op.cc:312] APPLY: finished
0623 08:04:35.501682 (+ 42us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.501711 (+ 29us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.501722 (+ 11us) write_op.cc:454] Released schema lock
0623 08:04:35.501727 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":33,"apply.run_cpu_time_us":294,"apply.run_wall_time_us":295,"num_ops":2,"prepare.queue_time_us":10717,"prepare.run_cpu_time_us":1493,"prepare.run_wall_time_us":35575,"replication_time_us":66591,"spinlock_wait_cycles":79363456}]]}
I20250623 08:04:35.503515 1366 consensus_queue.cc:1035] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:35.503756 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3540dde86f8442a89a59232aa4b53612" } modify_external_catalogs: true
I20250623 08:04:35.503974 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3540dde86f8442a89a59232aa4b53612" } modify_external_catalogs: true
I20250623 08:04:35.505682 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 789d1c20aa72480ab71123c799bbde4c
I20250623 08:04:35.506186 32058 catalog_manager.cc:6652] ReplaceTablet: tablet 789d1c20aa72480ab71123c799bbde4c deleted and replaced by tablet 0cca63991d5b4ef5bd8a9181d127ebe3
I20250623 08:04:35.506294 32058 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36116 (request call id 227) took 1054 ms. Trace:
I20250623 08:04:35.506400 32058 rpcz_store.cc:276] 0623 08:04:34.451496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.382804 (+931308us) service_pool.cc:225] Handling call
0623 08:04:35.403960 (+ 21156us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d080605b 00007f19c764f5d7 00007f19c764f305 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:35.506262 (+102302us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.469744 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.469853 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.469859 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:35.469862 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:35.469972 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.469982 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.469985 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.470054 (+ 69us) tablet.cc:699] Row locks acquired
0623 08:04:35.470056 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:35.470093 (+ 37us) write_op.cc:270] Start()
0623 08:04:35.470125 (+ 32us) write_op.cc:276] Timestamp: P: 1750665875470053 usec, L: 0
0623 08:04:35.470129 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.470270 (+ 141us) log.cc:844] Serialized 622 byte log entry
0623 08:04:35.501858 (+ 31588us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.501991 (+ 133us) write_op.cc:301] APPLY: starting
0623 08:04:35.502002 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.502034 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.502060 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.502182 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.502229 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:35.502236 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.502297 (+ 61us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.502375 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.502390 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:35.502395 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.502463 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.502468 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":47864192,"child_traces":[["op",{"apply.queue_time_us":94,"apply.run_cpu_time_us":461,"apply.run_wall_time_us":462,"num_ops":2,"prepare.queue_time_us":44664,"prepare.run_cpu_time_us":596,"prepare.run_wall_time_us":597,"replication_time_us":31710}]]}
I20250623 08:04:35.507246 32058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_303d632a02404fbe8b9baa353a8d3747" } new_table_name: "default.Table_e279e539bb974bd28995fc3110dffc56" modify_external_catalogs: true
I20250623 08:04:35.508801 952 ts_tablet_manager.cc:1918] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.509961 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 501b5233522f43c7aed3c9770adb5c03 with delete_type TABLET_DATA_DELETED (replaced by tablet 74c1cdc83ffd4a09a163479ac58fd405) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.510227 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 789d1c20aa72480ab71123c799bbde4c with delete_type TABLET_DATA_DELETED (replaced by tablet 0cca63991d5b4ef5bd8a9181d127ebe3) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.511332 342 consensus_queue.cc:260] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.512001 342 ts_tablet_manager.cc:1428] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
I20250623 08:04:35.512280 342 tablet_bootstrap.cc:492] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:35.513975 342 tablet_bootstrap.cc:654] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.514815 32056 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_28d324870b1049289c32fbfcf8c44ebf" } modify_external_catalogs: true
I20250623 08:04:35.515039 32056 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_28d324870b1049289c32fbfcf8c44ebf" } modify_external_catalogs: true
I20250623 08:04:35.517491 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 501b5233522f43c7aed3c9770adb5c03 with delete_type TABLET_DATA_DELETED (replaced by tablet 74c1cdc83ffd4a09a163479ac58fd405) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:35.517922 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 789d1c20aa72480ab71123c799bbde4c with delete_type TABLET_DATA_DELETED (replaced by tablet 0cca63991d5b4ef5bd8a9181d127ebe3) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:35.517872 32055 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_5c43b8626a6640e4a61e495ab2ea5697" } modify_external_catalogs: true
I20250623 08:04:35.518060 32055 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_5c43b8626a6640e4a61e495ab2ea5697" } modify_external_catalogs: true
I20250623 08:04:35.519542 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet db81250d1e164a7d94affd764fc0ff0c
I20250623 08:04:35.520136 32057 catalog_manager.cc:6652] ReplaceTablet: tablet db81250d1e164a7d94affd764fc0ff0c deleted and replaced by tablet 89a24019b797467e857b744d701fb1ed
I20250623 08:04:35.520529 32057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_5c6bbeaad8274e05904301c82a72aa74" } new_table_name: "default.Table_78fab2c6cc5c4523823de72525524e88" modify_external_catalogs: true
I20250623 08:04:35.522416 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet db81250d1e164a7d94affd764fc0ff0c with delete_type TABLET_DATA_DELETED (replaced by tablet 89a24019b797467e857b744d701fb1ed) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:35.523471 1412 tablet.cc:1722] T d02f74c24a174b18b08a870a4e47fefb P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:35.527103 1413 tablet.cc:1722] T d02f74c24a174b18b08a870a4e47fefb P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:35.535255 952 log.cc:1199] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/35ab9f77a81a4490abb0d2c0c69cd90f
I20250623 08:04:35.535943 952 ts_tablet_manager.cc:1939] T 35ab9f77a81a4490abb0d2c0c69cd90f P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:35.537894 339 tablet_bootstrap.cc:492] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:35.538084 339 ts_tablet_manager.cc:1397] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.121s user 0.005s sys 0.000s
I20250623 08:04:35.539156 339 raft_consensus.cc:357] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.539402 339 raft_consensus.cc:383] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.539498 339 raft_consensus.cc:738] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.539778 339 consensus_queue.cc:260] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.540215 339 ts_tablet_manager.cc:1428] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:35.540493 339 tablet_bootstrap.cc:492] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:35.541496 1423 tablet.cc:1722] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:35.542199 339 tablet_bootstrap.cc:654] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.546414 981 ts_tablet_manager.cc:1918] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250623 08:04:35.546638 981 log.cc:1199] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/8d8f38a6de1e45e7bed4abdced684c92
I20250623 08:04:35.547201 981 ts_tablet_manager.cc:1939] T 8d8f38a6de1e45e7bed4abdced684c92 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:35.547175 32453 consensus_peers.cc:487] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fdc9e53da264a5ca1f70e1846c906f0. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:35.500686 343 tablet_bootstrap.cc:492] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:35.509172 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 501b5233522f43c7aed3c9770adb5c03 with delete_type TABLET_DATA_DELETED (replaced by tablet 74c1cdc83ffd4a09a163479ac58fd405) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.509470 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 789d1c20aa72480ab71123c799bbde4c with delete_type TABLET_DATA_DELETED (replaced by tablet 0cca63991d5b4ef5bd8a9181d127ebe3) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.522114 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet db81250d1e164a7d94affd764fc0ff0c with delete_type TABLET_DATA_DELETED (replaced by tablet 89a24019b797467e857b744d701fb1ed) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.553125 343 ts_tablet_manager.cc:1397] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.002s
I20250623 08:04:35.540762 1278 raft_consensus.cc:491] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.553202 979 raft_consensus.cc:2241] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:35.553273 1278 raft_consensus.cc:513] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.553401 979 raft_consensus.cc:2270] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.549688 1418 tablet.cc:1722] T d02f74c24a174b18b08a870a4e47fefb P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:35.553906 1278 leader_election.cc:290] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:35.555437 1426 tablet.cc:1722] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:35.556285 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 52) took 4589 ms. Trace:
I20250623 08:04:35.556408 952 rpcz_store.cc:276] 0623 08:04:30.967095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.967160 (+ 65us) service_pool.cc:225] Handling call
0623 08:04:35.454786 (+4487626us) ts_tablet_manager.cc:1145] Deleting tablet 35ab9f77a81a4490abb0d2c0c69cd90f
0623 08:04:35.454796 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.505346 (+ 50550us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.508672 (+ 3326us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.556237 (+ 47565us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4487296}
I20250623 08:04:35.556890 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 35ab9f77a81a4490abb0d2c0c69cd90f (table default.Table_92e23e1081544f8c9b99f14654981494 [id=bbf8e66e745141348c56d1cea5716898]) successfully deleted
I20250623 08:04:35.557512 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b19e5f2359a84c7ea02a0b25240a54cb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.553906 343 raft_consensus.cc:357] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.557861 343 raft_consensus.cc:383] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.557915 343 raft_consensus.cc:738] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:35.557945 32589 leader_election.cc:343] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.558187 952 tablet_replica.cc:331] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:35.558094 343 consensus_queue.cc:260] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.558524 343 ts_tablet_manager.cc:1428] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:35.560034 979 ts_tablet_manager.cc:1905] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.563333 343 tablet_bootstrap.cc:492] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.564910 343 tablet_bootstrap.cc:654] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.565032 342 tablet_bootstrap.cc:492] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.565155 342 ts_tablet_manager.cc:1397] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.001s
I20250623 08:04:35.565893 342 raft_consensus.cc:357] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.566051 342 raft_consensus.cc:383] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.566102 342 raft_consensus.cc:738] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.566322 342 consensus_queue.cc:260] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.566936 342 ts_tablet_manager.cc:1428] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:35.548568 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet db81250d1e164a7d94affd764fc0ff0c with delete_type TABLET_DATA_DELETED (replaced by tablet 89a24019b797467e857b744d701fb1ed) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:35.554433 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b19e5f2359a84c7ea02a0b25240a54cb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.571872 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 83) took 3156 ms. Trace:
I20250623 08:04:35.573662 1413 tablet.cc:1722] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:35.576252 32589 leader_election.cc:343] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.576378 32589 leader_election.cc:304] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.576730 1278 raft_consensus.cc:2747] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.577201 981 rpcz_store.cc:276] 0623 08:04:32.415670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.415717 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:35.362033 (+2946316us) ts_tablet_manager.cc:1145] Deleting tablet 8d8f38a6de1e45e7bed4abdced684c92
0623 08:04:35.362046 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.449125 (+ 87079us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.546264 (+ 97139us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.571835 (+ 25571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2946034}
I20250623 08:04:35.577644 981 tablet_replica.cc:331] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:35.578660 981 raft_consensus.cc:2241] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.588224 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 4cf1f3943a3d49318c435921ebeaaf33 (DEFAULT_TABLE table=default.Table_39a59f2375e743109b067b2de8b6d4db [id=9125a8d179614084bebc3591092905e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:35.588423 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 4cf1f3943a3d49318c435921ebeaaf33 (DEFAULT_TABLE table=default.Table_39a59f2375e743109b067b2de8b6d4db [id=9125a8d179614084bebc3591092905e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:35.588495 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 8d8f38a6de1e45e7bed4abdced684c92 (table default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e [id=7a569a9fe3bd4c608ddca48a734df1da]) successfully deleted
I20250623 08:04:35.588856 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 077bb090b3ea4d029463304d7a6feee5 (DEFAULT_TABLE table=default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c [id=387b3fc3d26b457099bcbb3b8ef3ac9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:35.588879 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 077bb090b3ea4d029463304d7a6feee5 (DEFAULT_TABLE table=default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c [id=387b3fc3d26b457099bcbb3b8ef3ac9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:35.593806 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:35.594151 979 ts_tablet_manager.cc:1918] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:35.587744 32376 raft_consensus.cc:3058] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.594205 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:35.594604 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:35.589565 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 4cf1f3943a3d49318c435921ebeaaf33 (DEFAULT_TABLE table=default.Table_39a59f2375e743109b067b2de8b6d4db [id=9125a8d179614084bebc3591092905e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:35.595032 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:35.595108 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf1f3943a3d49318c435921ebeaaf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:35.595484 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 136e482995a24c5daec55a701f3091fa (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:35.595840 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:35.596267 32050 catalog_manager.cc:5582] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:35.597884 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:35.598124 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:35.598479 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:35.598856 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:35.599016 32050 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:35.590088 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 077bb090b3ea4d029463304d7a6feee5 (DEFAULT_TABLE table=default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c [id=387b3fc3d26b457099bcbb3b8ef3ac9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:35.600068 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077bb090b3ea4d029463304d7a6feee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:35.610213 848 consensus_queue.cc:1035] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:35.614385 746 raft_consensus.cc:491] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.615003 746 raft_consensus.cc:513] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.615701 746 leader_election.cc:290] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.618263 848 consensus_queue.cc:1035] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:35.622432 952 raft_consensus.cc:2241] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.634280 952 raft_consensus.cc:2270] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.635185 342 tablet_bootstrap.cc:492] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:35.636981 342 tablet_bootstrap.cc:654] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.639175 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf1f3943a3d49318c435921ebeaaf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:35.645756 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077bb090b3ea4d029463304d7a6feee5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:35.648084 32314 leader_election.cc:343] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: ea702abb31fb41498ab61bbb05b1f7e2
I20250623 08:04:35.649158 746 raft_consensus.cc:491] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:35.649333 746 raft_consensus.cc:513] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.650378 746 leader_election.cc:290] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.653797 981 raft_consensus.cc:2270] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.654426 981 ts_tablet_manager.cc:1905] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.659034 952 ts_tablet_manager.cc:1905] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:35.664933 32314 leader_election.cc:343] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: ea702abb31fb41498ab61bbb05b1f7e2
I20250623 08:04:35.665081 32314 leader_election.cc:304] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:35.665884 1417 raft_consensus.cc:2747] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.666350 342 tablet_bootstrap.cc:492] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.666484 342 ts_tablet_manager.cc:1397] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.003s
I20250623 08:04:35.667296 342 raft_consensus.cc:357] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.667474 342 raft_consensus.cc:383] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.667541 342 raft_consensus.cc:738] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.667754 342 consensus_queue.cc:260] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.668962 848 raft_consensus.cc:491] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.669071 848 raft_consensus.cc:513] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.669590 848 leader_election.cc:290] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.670552 848 consensus_queue.cc:1035] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:35.670682 342 ts_tablet_manager.cc:1428] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250623 08:04:35.671180 32453 consensus_peers.cc:487] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:35.671334 32453 consensus_peers.cc:487] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:35.671487 342 tablet_bootstrap.cc:492] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:35.671521 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0117cbb09dd4067836266c9d6a32adf" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.672595 1417 raft_consensus.cc:491] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:35.672884 1417 raft_consensus.cc:513] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.673803 1417 leader_election.cc:290] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.680687 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a070895f4984493a3b9f6fa779797a9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.682101 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "966a159e212040109e2842c15b69fb71" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.682258 32513 raft_consensus.cc:2466] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
W20250623 08:04:35.684311 32453 leader_election.cc:343] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.684514 342 tablet_bootstrap.cc:654] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:35.684810 32314 leader_election.cc:343] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: f0117cbb09dd4067836266c9d6a32adf
W20250623 08:04:35.685107 32314 leader_election.cc:343] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 966a159e212040109e2842c15b69fb71
I20250623 08:04:35.685505 32314 leader_election.cc:304] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:35.686071 1417 raft_consensus.cc:2747] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:35.689005 339 tablet_bootstrap.cc:492] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:35.689262 339 ts_tablet_manager.cc:1397] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.149s user 0.004s sys 0.001s
I20250623 08:04:35.690205 339 raft_consensus.cc:357] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.694525 952 ts_tablet_manager.cc:1918] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.607501 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf1f3943a3d49318c435921ebeaaf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:35.612936 343 tablet_bootstrap.cc:492] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:35.695112 979 log.cc:1199] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/70f9d21b8c144f9cada9bdbe0fbb1538
I20250623 08:04:35.695214 343 ts_tablet_manager.cc:1397] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.003s
I20250623 08:04:35.695487 979 ts_tablet_manager.cc:1939] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:35.695782 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 97) took 4204 ms. Trace:
I20250623 08:04:35.695864 979 rpcz_store.cc:276] 0623 08:04:31.490900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.491260 (+ 360us) service_pool.cc:225] Handling call
0623 08:04:35.475845 (+3984585us) ts_tablet_manager.cc:1145] Deleting tablet 70f9d21b8c144f9cada9bdbe0fbb1538
0623 08:04:35.475854 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.590733 (+114879us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.594068 (+ 3335us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.695767 (+101699us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3984296}
I20250623 08:04:35.696089 343 raft_consensus.cc:357] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.696239 343 raft_consensus.cc:383] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.696277 339 raft_consensus.cc:383] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.696295 343 raft_consensus.cc:738] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.696256 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 70f9d21b8c144f9cada9bdbe0fbb1538 (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]) successfully deleted
I20250623 08:04:35.696360 339 raft_consensus.cc:738] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.696488 343 consensus_queue.cc:260] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.696589 339 consensus_queue.cc:260] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.647881 1278 raft_consensus.cc:491] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.696863 1278 raft_consensus.cc:513] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.697167 339 ts_tablet_manager.cc:1428] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250623 08:04:35.697468 339 tablet_bootstrap.cc:492] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:35.697433 1278 leader_election.cc:290] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:35.653155 1128 consensus_queue.cc:1035] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:35.698457 1426 tablet.cc:1722] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:35.675813 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a070895f4984493a3b9f6fa779797a9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.698853 32646 raft_consensus.cc:2466] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:35.699194 339 tablet_bootstrap.cc:654] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.690799 32314 leader_election.cc:304] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:35.700193 1417 raft_consensus.cc:2802] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:35.700305 1417 raft_consensus.cc:491] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:35.700399 1417 raft_consensus.cc:3058] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:35.702718 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077bb090b3ea4d029463304d7a6feee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:35.704869 1417 raft_consensus.cc:513] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.705688 1417 leader_election.cc:290] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.699679 32453 leader_election.cc:304] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.706391 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "966a159e212040109e2842c15b69fb71" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:35.706545 32513 raft_consensus.cc:3058] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:35.706960 848 raft_consensus.cc:2802] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.707033 848 raft_consensus.cc:491] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.707093 848 raft_consensus.cc:3058] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.707659 657 consensus_queue.cc:1035] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:35.708352 952 log.cc:1199] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/4ea81642050141a9be84f151a5785abc
I20250623 08:04:35.708768 952 ts_tablet_manager.cc:1939] T 4ea81642050141a9be84f151a5785abc P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:35.709072 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 55) took 4741 ms. Trace:
I20250623 08:04:35.709138 952 rpcz_store.cc:276] 0623 08:04:30.967479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.968504 (+ 1025us) service_pool.cc:225] Handling call
0623 08:04:35.557151 (+4588647us) ts_tablet_manager.cc:1145] Deleting tablet 4ea81642050141a9be84f151a5785abc
0623 08:04:35.558125 (+ 974us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.662567 (+104442us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.694406 (+ 31839us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.709059 (+ 14653us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4588354}
I20250623 08:04:35.709560 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 4ea81642050141a9be84f151a5785abc (table default.Table_176ee9ee2de7444ca3c54ee24e3e5667 [id=1475deb3ccfa49d5935900afa29dee6f]) successfully deleted
I20250623 08:04:35.709692 32513 raft_consensus.cc:2466] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 2.
I20250623 08:04:35.709800 848 raft_consensus.cc:513] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.710350 848 leader_election.cc:290] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.710716 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a070895f4984493a3b9f6fa779797a9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.710898 32650 raft_consensus.cc:3058] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.710966 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a070895f4984493a3b9f6fa779797a9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:35.711459 32314 leader_election.cc:304] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:35.711867 32453 leader_election.cc:343] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.712199 32314 leader_election.cc:343] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 966a159e212040109e2842c15b69fb71
I20250623 08:04:35.713444 32650 raft_consensus.cc:2466] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.713781 32453 leader_election.cc:304] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.714115 657 raft_consensus.cc:2802] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.714211 657 raft_consensus.cc:695] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.714432 657 consensus_queue.cc:237] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.716269 342 tablet_bootstrap.cc:492] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.716398 342 ts_tablet_manager.cc:1397] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20250623 08:04:35.717069 342 raft_consensus.cc:357] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.717217 342 raft_consensus.cc:383] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.717278 342 raft_consensus.cc:738] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.717478 342 consensus_queue.cc:260] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.717969 342 ts_tablet_manager.cc:1428] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:35.718484 342 tablet_bootstrap.cc:492] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:35.720306 342 tablet_bootstrap.cc:654] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.728807 1417 raft_consensus.cc:2802] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:35.729167 1417 raft_consensus.cc:695] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:35.729579 1417 consensus_queue.cc:237] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.730278 343 ts_tablet_manager.cc:1428] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.035s user 0.003s sys 0.008s
I20250623 08:04:35.733690 981 ts_tablet_manager.cc:1918] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250623 08:04:35.733907 981 log.cc:1199] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/d0ee5a4fe03848229524cc0105226d8c
I20250623 08:04:35.739980 979 tablet_replica.cc:331] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:35.740221 979 raft_consensus.cc:2241] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.740537 979 raft_consensus.cc:2270] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.741117 979 ts_tablet_manager.cc:1905] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.742206 952 tablet_replica.cc:331] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:35.742771 952 raft_consensus.cc:2241] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.743608 343 tablet_bootstrap.cc:492] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.744764 952 raft_consensus.cc:2270] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.746526 1443 raft_consensus.cc:491] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.746740 1443 raft_consensus.cc:513] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.747466 1443 leader_election.cc:290] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:35.746380 343 tablet_bootstrap.cc:654] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.750195 952 ts_tablet_manager.cc:1905] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.754632 339 tablet_bootstrap.cc:492] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:35.754848 339 ts_tablet_manager.cc:1397] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20250623 08:04:35.755431 342 tablet_bootstrap.cc:492] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.756767 343 tablet_bootstrap.cc:492] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
W20250623 08:04:35.759159 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 89007.3us
I20250623 08:04:35.759084 343 ts_tablet_manager.cc:1397] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250623 08:04:35.760188 343 raft_consensus.cc:357] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:35.760455 343 raft_consensus.cc:383] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.760563 343 raft_consensus.cc:738] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.760849 343 consensus_queue.cc:260] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:35.761504 343 ts_tablet_manager.cc:1428] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:35.762532 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a0bbafe1ea34ab5ae1b3de25b50d4a3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.763247 343 tablet_bootstrap.cc:492] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.765249 343 tablet_bootstrap.cc:654] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.767199 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75a6ebe0d6b0463b8ddcb1a091a2cc6f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:35.769930 32589 leader_election.cc:343] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.770663 32589 consensus_peers.cc:487] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:35.771044 981 ts_tablet_manager.cc:1939] T d0ee5a4fe03848229524cc0105226d8c P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:35.772112 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 82) took 3356 ms. Trace:
W20250623 08:04:35.772040 32589 leader_election.cc:343] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.775187 343 tablet_bootstrap.cc:492] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:35.775363 343 ts_tablet_manager.cc:1397] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20250623 08:04:35.774868 1412 tablet.cc:1722] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:35.778919 343 raft_consensus.cc:357] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:35.779153 343 raft_consensus.cc:383] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.779407 343 raft_consensus.cc:738] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.779716 343 consensus_queue.cc:260] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:35.780244 343 ts_tablet_manager.cc:1428] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250623 08:04:35.780551 343 tablet_bootstrap.cc:492] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.782172 343 tablet_bootstrap.cc:654] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.775558 32428 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:35.775933 339 raft_consensus.cc:357] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.786841 339 raft_consensus.cc:383] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.787338 339 raft_consensus.cc:738] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.788252 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:35.787012 981 rpcz_store.cc:276] 0623 08:04:32.415546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.416026 (+ 480us) service_pool.cc:225] Handling call
0623 08:04:35.577572 (+3161546us) ts_tablet_manager.cc:1145] Deleting tablet d0ee5a4fe03848229524cc0105226d8c
0623 08:04:35.577582 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.657980 (+ 80398us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.733546 (+ 75566us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.772043 (+ 38497us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24832,"tablet-delete.queue_time_us":3161341}
I20250623 08:04:35.789822 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 85) took 3368 ms. Trace:
I20250623 08:04:35.790019 981 rpcz_store.cc:276] 0623 08:04:32.421459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.421504 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:35.789612 (+3368108us) ts_tablet_manager.cc:1145] Deleting tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3
0623 08:04:35.789623 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.789756 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3367879}
I20250623 08:04:35.792657 343 tablet_bootstrap.cc:492] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:35.792857 343 ts_tablet_manager.cc:1397] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250623 08:04:35.788872 339 consensus_queue.cc:260] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.795255 343 raft_consensus.cc:357] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:35.795461 343 raft_consensus.cc:383] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.795554 343 raft_consensus.cc:738] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.795828 343 consensus_queue.cc:260] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:35.797271 339 ts_tablet_manager.cc:1428] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20250623 08:04:35.797592 339 tablet_bootstrap.cc:492] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:35.789314 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet d0ee5a4fe03848229524cc0105226d8c (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]) successfully deleted
W20250623 08:04:35.799420 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 already in progress: creating tablet
I20250623 08:04:35.799845 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:35.801059 343 ts_tablet_manager.cc:1428] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250623 08:04:35.801328 343 tablet_bootstrap.cc:492] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.802973 343 tablet_bootstrap.cc:654] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.803442 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 84) took 3382 ms. Trace:
I20250623 08:04:35.805677 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:35.806114 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_32db9f68dd974fb8aff372a8eeccce3d [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 1 got 0
W20250623 08:04:35.756297 32453 consensus_peers.cc:487] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:35.806962 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 50687.8us
I20250623 08:04:35.810786 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a0bbafe1ea34ab5ae1b3de25b50d4a3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.811882 848 raft_consensus.cc:491] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.812005 848 raft_consensus.cc:513] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.812584 848 leader_election.cc:290] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.810926 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75a6ebe0d6b0463b8ddcb1a091a2cc6f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.813712 1413 tablet.cc:1722] T 6f9e0d1b85a94ae9ae86e53688a55db5 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:35.814170 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "136e482995a24c5daec55a701f3091fa" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.814411 32646 raft_consensus.cc:2466] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:35.815187 32453 leader_election.cc:304] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:35.816042 1359 raft_consensus.cc:2802] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.816139 1359 raft_consensus.cc:491] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.816223 1359 raft_consensus.cc:3058] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.819415 1447 raft_consensus.cc:491] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.819540 1447 raft_consensus.cc:513] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.820112 1447 leader_election.cc:290] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.820730 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c58231f67604422a1497e4c1b44ec52" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.820894 32656 raft_consensus.cc:2466] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:35.821300 32453 leader_election.cc:304] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:35.821575 1447 raft_consensus.cc:2802] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.821681 1447 raft_consensus.cc:491] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.821743 1447 raft_consensus.cc:3058] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.824590 1447 raft_consensus.cc:513] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.825188 1447 leader_election.cc:290] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.826061 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c58231f67604422a1497e4c1b44ec52" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.826377 32656 raft_consensus.cc:3058] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.828935 32656 raft_consensus.cc:2466] T 0c58231f67604422a1497e4c1b44ec52 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.829396 32453 leader_election.cc:304] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:35.829875 1447 raft_consensus.cc:2802] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.830085 1447 raft_consensus.cc:695] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.830466 1447 consensus_queue.cc:237] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:35.832307 32589 leader_election.cc:343] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.832502 1359 raft_consensus.cc:513] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.832765 32589 leader_election.cc:304] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.806967 342 ts_tablet_manager.cc:1397] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.000s
I20250623 08:04:35.833923 1440 raft_consensus.cc:2747] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.822155 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "136e482995a24c5daec55a701f3091fa" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.822949 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c58231f67604422a1497e4c1b44ec52" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.825934 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c58231f67604422a1497e4c1b44ec52" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:35.835721 32453 leader_election.cc:343] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.835793 32589 leader_election.cc:343] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.836059 32453 leader_election.cc:343] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.836385 32453 leader_election.cc:343] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.836138 32589 leader_election.cc:304] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.836771 981 rpcz_store.cc:276] 0623 08:04:32.421333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.421805 (+ 472us) service_pool.cc:225] Handling call
0623 08:04:35.803260 (+3381455us) ts_tablet_manager.cc:1145] Deleting tablet 205da880685b49b3b8669f73dc2c563b
0623 08:04:35.803269 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.803383 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3381118}
I20250623 08:04:35.837652 1366 raft_consensus.cc:491] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.837765 1366 raft_consensus.cc:513] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.838241 1366 leader_election.cc:290] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.839210 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.839421 32656 raft_consensus.cc:2466] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:35.839452 979 ts_tablet_manager.cc:1918] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.839598 979 log.cc:1199] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/38ecd4c243b840f4b66d8e01078dca64
I20250623 08:04:35.839941 32453 leader_election.cc:304] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:35.840059 979 ts_tablet_manager.cc:1939] T 38ecd4c243b840f4b66d8e01078dca64 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:35.840301 1159 raft_consensus.cc:2802] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.840408 1159 raft_consensus.cc:491] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.840514 1159 raft_consensus.cc:3058] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.840708 1366 raft_consensus.cc:491] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.840832 1366 raft_consensus.cc:513] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.841342 1366 leader_election.cc:290] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.843923 1159 raft_consensus.cc:513] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.844748 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c33fa0a7ff34d2985c5e07fcf88eaa1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.844952 32656 raft_consensus.cc:2466] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:35.845816 1440 raft_consensus.cc:2747] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.838119 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "136e482995a24c5daec55a701f3091fa" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:35.838987 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.844493 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c33fa0a7ff34d2985c5e07fcf88eaa1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:35.848829 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 205da880685b49b3b8669f73dc2c563b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 205da880685b49b3b8669f73dc2c563b already in progress: creating tablet
I20250623 08:04:35.850396 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 87) took 3425 ms. Trace:
I20250623 08:04:35.850873 1359 leader_election.cc:290] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.851677 981 rpcz_store.cc:276] 0623 08:04:32.425162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.425205 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:35.850183 (+3424978us) ts_tablet_manager.cc:1145] Deleting tablet e9f8aa424bbb4aa8a907259270d46656
0623 08:04:35.850197 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.850321 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3424788}
W20250623 08:04:35.852623 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet e9f8aa424bbb4aa8a907259270d46656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9f8aa424bbb4aa8a907259270d46656 already in progress: creating tablet
I20250623 08:04:35.852638 1159 leader_election.cc:290] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.853391 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 86) took 3428 ms. Trace:
I20250623 08:04:35.844732 952 ts_tablet_manager.cc:1918] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.854851 952 log.cc:1199] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/e22f9bae5f7248af99cec06ce6dbbd41
I20250623 08:04:35.854880 1159 raft_consensus.cc:491] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.855257 952 ts_tablet_manager.cc:1939] T e22f9bae5f7248af99cec06ce6dbbd41 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:35.855074 1159 raft_consensus.cc:513] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.855566 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 56) took 4888 ms. Trace:
I20250623 08:04:35.855661 952 rpcz_store.cc:276] 0623 08:04:30.967597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.968952 (+ 1355us) service_pool.cc:225] Handling call
0623 08:04:35.742165 (+4773213us) ts_tablet_manager.cc:1145] Deleting tablet e22f9bae5f7248af99cec06ce6dbbd41
0623 08:04:35.742174 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.826271 (+ 84097us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.844631 (+ 18360us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.855551 (+ 10920us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4772994}
I20250623 08:04:35.856015 952 tablet_replica.cc:331] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:35.856233 952 raft_consensus.cc:2241] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.855557 981 rpcz_store.cc:276] 0623 08:04:32.425051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.425554 (+ 503us) service_pool.cc:225] Handling call
0623 08:04:35.853181 (+3427627us) ts_tablet_manager.cc:1145] Deleting tablet 8dc696236a15420e9666cca708068fb2
0623 08:04:35.853196 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.853316 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3427446}
I20250623 08:04:35.845378 32453 leader_election.cc:304] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:35.856542 1159 leader_election.cc:290] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:35.856918 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 8dc696236a15420e9666cca708068fb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dc696236a15420e9666cca708068fb2 already in progress: creating tablet
I20250623 08:04:35.857055 1159 raft_consensus.cc:2802] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.857331 1159 raft_consensus.cc:491] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.857635 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.857838 32646 raft_consensus.cc:3058] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.857796 1159 raft_consensus.cc:3058] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.858006 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a414144276f6484ea4690204752e2142" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.858181 32650 raft_consensus.cc:2466] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:35.860944 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:35.861869 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet e22f9bae5f7248af99cec06ce6dbbd41 (table default.Table_dfc3da325f6849bfba88293a5eabb3e4 [id=ad93708e20ae4ef087d9a9b028d89806]) successfully deleted
I20250623 08:04:35.862628 32646 raft_consensus.cc:2466] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:35.862988 32453 leader_election.cc:343] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.863205 32453 leader_election.cc:343] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.863320 1159 raft_consensus.cc:513] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.863520 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:35.863484 32453 leader_election.cc:343] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.864073 32453 leader_election.cc:304] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:35.864455 848 raft_consensus.cc:2802] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.864413 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a414144276f6484ea4690204752e2142" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.864518 848 raft_consensus.cc:491] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.864568 848 raft_consensus.cc:3058] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.864698 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "136e482995a24c5daec55a701f3091fa" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.864921 32656 raft_consensus.cc:3058] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.865124 339 tablet_bootstrap.cc:654] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.866536 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 88) took 3441 ms. Trace:
I20250623 08:04:35.867442 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:35.867617 32656 raft_consensus.cc:2466] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.867681 848 raft_consensus.cc:513] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.868144 981 rpcz_store.cc:276] 0623 08:04:32.425516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.427389 (+ 1873us) service_pool.cc:225] Handling call
0623 08:04:35.866327 (+3438938us) ts_tablet_manager.cc:1145] Deleting tablet ecb422550efd40cf82bc92fca5c25eda
0623 08:04:35.866339 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.866472 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3438752}
I20250623 08:04:35.868678 848 leader_election.cc:290] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.869527 32453 leader_election.cc:304] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:35.869957 848 raft_consensus.cc:2802] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.870167 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 89) took 3442 ms. Trace:
W20250623 08:04:35.869073 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet ecb422550efd40cf82bc92fca5c25eda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb422550efd40cf82bc92fca5c25eda already in progress: creating tablet
I20250623 08:04:35.870309 848 raft_consensus.cc:695] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.870739 848 consensus_queue.cc:237] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.871382 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:35.871799 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 98) took 4349 ms. Trace:
I20250623 08:04:35.871999 981 rpcz_store.cc:276] 0623 08:04:32.427716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.427757 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:35.869975 (+3442218us) ts_tablet_manager.cc:1145] Deleting tablet a414144276f6484ea4690204752e2142
0623 08:04:35.869985 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.870104 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3442042}
I20250623 08:04:35.872277 32453 leader_election.cc:304] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.872660 657 raft_consensus.cc:2802] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.872761 657 raft_consensus.cc:695] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.872953 657 consensus_queue.cc:237] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.872400 979 rpcz_store.cc:276] 0623 08:04:31.491065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.491590 (+ 525us) service_pool.cc:225] Handling call
0623 08:04:35.730855 (+4239265us) ts_tablet_manager.cc:1145] Deleting tablet 38ecd4c243b840f4b66d8e01078dca64
0623 08:04:35.739917 (+ 9062us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.744890 (+ 4973us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.839310 (+ 94420us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.840883 (+ 1573us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4239035}
I20250623 08:04:35.873260 979 tablet_replica.cc:331] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:35.849846 342 raft_consensus.cc:357] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.874818 342 raft_consensus.cc:383] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.874886 342 raft_consensus.cc:738] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.875150 981 tablet_replica.cc:331] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:35.875069 342 consensus_queue.cc:260] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.875589 342 ts_tablet_manager.cc:1428] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.042s user 0.001s sys 0.000s
I20250623 08:04:35.875874 342 tablet_bootstrap.cc:492] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:35.877158 32453 leader_election.cc:343] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.877650 342 tablet_bootstrap.cc:654] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.877414 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a414144276f6484ea4690204752e2142" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.877683 1159 leader_election.cc:290] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.875591 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a414144276f6484ea4690204752e2142" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:35.878234 32656 raft_consensus.cc:3058] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.873397 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 38ecd4c243b840f4b66d8e01078dca64 (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]) successfully deleted
W20250623 08:04:35.878564 32453 leader_election.cc:343] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.878883 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet a414144276f6484ea4690204752e2142 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a414144276f6484ea4690204752e2142 already in progress: creating tablet
I20250623 08:04:35.879508 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]). Expected version 1 got 0
I20250623 08:04:35.879858 981 raft_consensus.cc:2241] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:35.873466 979 raft_consensus.cc:2241] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:35.880424 979 raft_consensus.cc:2270] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.856554 952 raft_consensus.cc:2270] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.880913 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c33fa0a7ff34d2985c5e07fcf88eaa1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:35.881933 981 raft_consensus.cc:2270] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:35.882406 1159 consensus_queue.cc:1035] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:35.882968 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:35.883078 952 ts_tablet_manager.cc:1905] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:35.883201 32453 leader_election.cc:343] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:35.883860 32453 leader_election.cc:343] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.881071 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c33fa0a7ff34d2985c5e07fcf88eaa1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.882546 1440 raft_consensus.cc:491] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.884302 32646 raft_consensus.cc:3058] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.884323 1440 raft_consensus.cc:513] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:35.884964 1440 leader_election.cc:290] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:35.885048 979 ts_tablet_manager.cc:1905] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:35.882174 339 tablet_bootstrap.cc:492] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:35.885792 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71fadb10e6134478b013358bba056389" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.887061 32646 raft_consensus.cc:2466] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:35.887269 32589 leader_election.cc:343] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.887708 339 ts_tablet_manager.cc:1397] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
I20250623 08:04:35.888842 339 raft_consensus.cc:357] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.889356 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet d5cd316e6f4447be958611ab07ed31fc (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:35.895676 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71fadb10e6134478b013358bba056389" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:35.895248 32453 leader_election.cc:304] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:35.896997 32589 leader_election.cc:343] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.897102 32589 leader_election.cc:304] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.897735 657 raft_consensus.cc:2802] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.897779 1278 raft_consensus.cc:2747] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.897845 657 raft_consensus.cc:695] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.898067 657 consensus_queue.cc:237] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.899474 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]). Expected version 3 got 0
I20250623 08:04:35.899977 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:35.900153 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:35.900907 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:35.901261 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_049497d4290c4bdfaf1e609b8b2ac9ac [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 2 got 0
I20250623 08:04:35.901691 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:35.902040 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 136e482995a24c5daec55a701f3091fa (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:35.902491 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f83f68ebda7e4826bbb17278cd4ac5a8 (table default.Table_6acb69304c424830a635f22f12343b31 [id=c1782341d0d34662889cff00d557ff81]). Expected version 1 got 0
I20250623 08:04:35.902698 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:35.903100 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:35.903424 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:35.903920 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:35.904196 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:35.904655 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 771ecfc41e2b4007a4d213cec5b128c6 (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:35.905100 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_aaa3ca8d9bb0406981f0dfff8cbf9e3c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 4 got 0
I20250623 08:04:35.905365 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:35.905879 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:35.906279 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b84b8b8713774ab4896842feb9fb075e (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:35.912111 339 raft_consensus.cc:383] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.912264 339 raft_consensus.cc:738] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.912642 339 consensus_queue.cc:260] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.899008 981 ts_tablet_manager.cc:1905] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:35.920821 32453 consensus_peers.cc:487] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3d32798168345edb9898751d083057c. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:35.926159 1413 tablet.cc:1722] T 712abfca1fa4489f882534c33b9db1be P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:35.927358 1426 tablet.cc:1722] T 712abfca1fa4489f882534c33b9db1be P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:35.935375 342 tablet_bootstrap.cc:492] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.935583 342 ts_tablet_manager.cc:1397] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.000s
I20250623 08:04:35.936035 339 ts_tablet_manager.cc:1428] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20250623 08:04:35.936431 342 raft_consensus.cc:357] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.936635 342 raft_consensus.cc:383] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.936704 342 raft_consensus.cc:738] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.936929 342 consensus_queue.cc:260] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.936570 1359 raft_consensus.cc:491] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.937218 1359 raft_consensus.cc:513] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.937887 1359 leader_election.cc:290] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.937865 1417 raft_consensus.cc:491] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:35.938063 1417 raft_consensus.cc:513] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.938936 1417 leader_election.cc:290] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.939899 342 ts_tablet_manager.cc:1428] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:35.940034 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025365d2fa0a43dd983d395a05cc2af1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:35.940160 342 tablet_bootstrap.cc:492] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:35.940266 32655 raft_consensus.cc:2466] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:35.941689 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025365d2fa0a43dd983d395a05cc2af1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:35.942027 342 tablet_bootstrap.cc:654] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:35.942742 32314 leader_election.cc:343] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 2ecf4af9e476495683bd63cac20d4672
W20250623 08:04:35.943687 32314 leader_election.cc:343] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 2ecf4af9e476495683bd63cac20d4672
W20250623 08:04:35.943843 32453 leader_election.cc:343] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.943885 32314 leader_election.cc:304] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
W20250623 08:04:35.944036 32453 consensus_peers.cc:487] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:35.944455 32453 leader_election.cc:304] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.944610 1417 raft_consensus.cc:2747] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:35.944700 1159 raft_consensus.cc:2802] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:35.944763 1159 raft_consensus.cc:491] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:35.944823 1159 raft_consensus.cc:3058] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.948207 32656 raft_consensus.cc:2466] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.948767 1159 raft_consensus.cc:513] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.949532 1159 leader_election.cc:290] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:35.950290 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025365d2fa0a43dd983d395a05cc2af1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:35.951135 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025365d2fa0a43dd983d395a05cc2af1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:35.951295 32655 raft_consensus.cc:3058] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:35.952965 979 ts_tablet_manager.cc:1918] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:35.953059 979 log.cc:1199] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/75a6ebe0d6b0463b8ddcb1a091a2cc6f
I20250623 08:04:35.953336 979 ts_tablet_manager.cc:1939] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:35.953915 32655 raft_consensus.cc:2466] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:35.954178 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 99) took 4462 ms. Trace:
I20250623 08:04:35.954303 979 rpcz_store.cc:276] 0623 08:04:31.491204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.491849 (+ 645us) service_pool.cc:225] Handling call
0623 08:04:35.873213 (+4381364us) ts_tablet_manager.cc:1145] Deleting tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f
0623 08:04:35.873224 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.949649 (+ 76425us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.952875 (+ 3226us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.954152 (+ 1277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4381184}
I20250623 08:04:35.954974 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]) successfully deleted
W20250623 08:04:35.955240 32453 leader_election.cc:343] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:35.955515 32453 leader_election.cc:304] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:35.955790 1359 raft_consensus.cc:2802] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:35.955957 1359 raft_consensus.cc:695] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:35.956231 1359 consensus_queue.cc:237] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.956612 343 tablet_bootstrap.cc:492] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:35.956790 343 ts_tablet_manager.cc:1397] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.156s user 0.048s sys 0.075s
I20250623 08:04:35.957573 343 raft_consensus.cc:357] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:35.957811 343 raft_consensus.cc:383] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.957906 343 raft_consensus.cc:738] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.958153 343 consensus_queue.cc:260] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:35.958693 343 ts_tablet_manager.cc:1428] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:35.958999 339 tablet_bootstrap.cc:492] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:35.959007 343 tablet_bootstrap.cc:492] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:35.960739 343 tablet_bootstrap.cc:654] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.961272 339 tablet_bootstrap.cc:654] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.972136 339 tablet_bootstrap.cc:492] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:35.972306 339 ts_tablet_manager.cc:1397] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250623 08:04:35.973153 339 raft_consensus.cc:357] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.973349 339 raft_consensus.cc:383] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:35.973456 339 raft_consensus.cc:738] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:35.974642 339 consensus_queue.cc:260] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:35.977257 339 ts_tablet_manager.cc:1428] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:35.960273 952 ts_tablet_manager.cc:1918] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.979190 952 log.cc:1199] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/776b412b3f564a7581c6a5e1c84e8499
I20250623 08:04:35.979247 339 tablet_bootstrap.cc:492] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:35.979101 848 consensus_queue.cc:1035] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:35.979933 952 ts_tablet_manager.cc:1939] T 776b412b3f564a7581c6a5e1c84e8499 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:35.980247 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 57) took 5010 ms. Trace:
I20250623 08:04:35.980326 952 rpcz_store.cc:276] 0623 08:04:30.969359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.969660 (+ 301us) service_pool.cc:225] Handling call
0623 08:04:35.855966 (+4886306us) ts_tablet_manager.cc:1145] Deleting tablet 776b412b3f564a7581c6a5e1c84e8499
0623 08:04:35.855977 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.886619 (+ 30642us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.960168 (+ 73549us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.980232 (+ 20064us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4886130}
I20250623 08:04:35.980638 952 tablet_replica.cc:331] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:35.980837 952 raft_consensus.cc:2241] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:35.981009 339 tablet_bootstrap.cc:654] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:35.984423 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:35.989091 32052 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_bc1251ce708a4467b5b1c71a1df307e0 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 1 got 0
I20250623 08:04:35.990324 981 ts_tablet_manager.cc:1918] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.990944 32055 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet d19d468ce13d4108a9badc36fc503306
I20250623 08:04:35.991539 32055 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 231) took 1373 ms. Trace:
I20250623 08:04:35.991655 32055 rpcz_store.cc:276] 0623 08:04:34.617900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.486580 (+868680us) service_pool.cc:225] Handling call
0623 08:04:35.517812 (+ 31232us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fa306 00007f19c7644956 00007f19c7644685 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:35.518168 (+ 356us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:35.518307 (+ 139us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:35.518373 (+ 66us) catalog_manager.cc:2799] Locking tablets
0623 08:04:35.603193 (+ 84820us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:35.620148 (+ 16955us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:35.620233 (+ 85us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:35.990897 (+370664us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:35.991474 (+ 577us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.605354 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.605441 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.605445 (+ 4us) write_op.cc:435] Acquired schema lock
0623 08:04:35.605447 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:35.605528 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.605538 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.605543 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.605645 (+ 102us) tablet.cc:699] Row locks acquired
0623 08:04:35.605650 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:35.605680 (+ 30us) write_op.cc:270] Start()
0623 08:04:35.605706 (+ 26us) write_op.cc:276] Timestamp: P: 1750665875605627 usec, L: 0
0623 08:04:35.605709 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.605859 (+ 150us) log.cc:844] Serialized 725 byte log entry
0623 08:04:35.618640 (+ 12781us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.619798 (+ 1158us) write_op.cc:301] APPLY: starting
0623 08:04:35.619806 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.619829 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.619845 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.619892 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.619928 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:35.619939 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:35.620000 (+ 61us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.620046 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.620062 (+ 16us) write_op.cc:454] Released schema lock
0623 08:04:35.620070 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.620132 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.620136 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":71218432,"child_traces":[["op",{"apply.queue_time_us":1127,"apply.run_cpu_time_us":329,"apply.run_wall_time_us":328,"num_ops":2,"prepare.queue_time_us":1565,"prepare.run_cpu_time_us":549,"prepare.run_wall_time_us":549,"replication_time_us":12919}]]}
I20250623 08:04:35.992422 32055 master_service.cc:948] ReplaceTablet: received request to replace tablet 70f9d21b8c144f9cada9bdbe0fbb1538 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:35.993652 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 18ea63263b9b466e922169eecafab747
I20250623 08:04:35.993655 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet d19d468ce13d4108a9badc36fc503306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:35.994169 32054 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 233) took 1370 ms. Trace:
I20250623 08:04:35.994256 32054 rpcz_store.cc:276] 0623 08:04:34.623609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.503565 (+879956us) service_pool.cc:225] Handling call
0623 08:04:35.504081 (+ 516us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:35.504254 (+ 173us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:35.504333 (+ 79us) catalog_manager.cc:2799] Locking tablets
0623 08:04:35.603941 (+ 99608us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:35.620490 (+ 16549us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:35.620561 (+ 71us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:35.993593 (+373032us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:35.994117 (+ 524us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.605939 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.606062 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.606069 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:35.606072 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:35.606165 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.606175 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.606182 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.606280 (+ 98us) tablet.cc:699] Row locks acquired
0623 08:04:35.606282 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:35.606308 (+ 26us) write_op.cc:270] Start()
0623 08:04:35.606331 (+ 23us) write_op.cc:276] Timestamp: P: 1750665875606257 usec, L: 0
0623 08:04:35.606334 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.606475 (+ 141us) log.cc:844] Serialized 756 byte log entry
0623 08:04:35.618676 (+ 12201us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.620189 (+ 1513us) write_op.cc:301] APPLY: starting
0623 08:04:35.620198 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.620224 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.620251 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.620293 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.620316 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:35.620323 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.620377 (+ 54us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.620409 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.620422 (+ 13us) write_op.cc:454] Released schema lock
0623 08:04:35.620427 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1486,"apply.run_cpu_time_us":291,"apply.run_wall_time_us":290,"num_ops":2,"prepare.queue_time_us":1451,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":581,"replication_time_us":12330,"thread_start_us":32884,"threads_started":1}]]}
I20250623 08:04:35.994908 32054 master_service.cc:948] ReplaceTablet: received request to replace tablet 16331ba6a34942a5a1b4ff4ac7bac55e from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:35.996318 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 18ea63263b9b466e922169eecafab747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:35.996526 32058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e898ede64e641519c8ee3ea1853a4b0 consensus state has no leader
I20250623 08:04:35.996699 32058 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 234) took 1292 ms. Trace:
I20250623 08:04:35.997115 32056 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 89bb7ea31b4d4ce7a3c063853142fbb2
I20250623 08:04:35.997258 32051 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 26f4a1b472e848ccb9a1ec71c1dd3516
I20250623 08:04:35.997594 32056 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 232) took 1374 ms. Trace:
I20250623 08:04:35.997681 32051 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 229) took 1423 ms. Trace:
I20250623 08:04:35.997785 32051 rpcz_store.cc:276] 0623 08:04:34.574316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.413222 (+838906us) service_pool.cc:225] Handling call
0623 08:04:35.471012 (+ 57790us) spinlock_profiling.cc:243] Waited 57.5 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fa306 00007f19c7644956 00007f19c7644685 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:35.472877 (+ 1865us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:35.473073 (+ 196us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:35.473160 (+ 87us) catalog_manager.cc:2799] Locking tablets
0623 08:04:35.473328 (+ 168us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:35.522720 (+ 49392us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:35.522842 (+ 122us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:35.997202 (+474360us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:35.997642 (+ 440us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.475406 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.475517 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.475525 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:35.475529 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:35.475666 (+ 137us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.475679 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.475686 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.475798 (+ 112us) tablet.cc:699] Row locks acquired
0623 08:04:35.475801 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:35.475830 (+ 29us) write_op.cc:270] Start()
0623 08:04:35.475855 (+ 25us) write_op.cc:276] Timestamp: P: 1750665875475791 usec, L: 0
0623 08:04:35.475859 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.476070 (+ 211us) log.cc:844] Serialized 759 byte log entry
0623 08:04:35.503115 (+ 27045us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.503183 (+ 68us) write_op.cc:301] APPLY: starting
0623 08:04:35.503194 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.503223 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.503245 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.503292 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.503317 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:35.503323 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:35.503391 (+ 68us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.503452 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.503467 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:35.503472 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":132301056,"child_traces":[["op",{"apply.queue_time_us":32,"apply.run_cpu_time_us":341,"apply.run_wall_time_us":341,"num_ops":2,"prepare.queue_time_us":1378,"prepare.run_cpu_time_us":705,"prepare.run_wall_time_us":1098,"replication_time_us":27237}]]}
I20250623 08:04:35.998560 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_af3136dbadfb43c892a197cc3a5d684e" } new_table_name: "default.Table_9aeb558e2ae64fe2b45bef5d2b6a2625" modify_external_catalogs: true
W20250623 08:04:36.001101 32053 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 025365d2fa0a43dd983d395a05cc2af1 consensus state has no leader
I20250623 08:04:36.001286 32053 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 228) took 1427 ms. Trace:
I20250623 08:04:36.001422 952 raft_consensus.cc:2270] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.001998 952 ts_tablet_manager.cc:1905] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.001405 32053 rpcz_store.cc:276] 0623 08:04:34.574197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.384802 (+810605us) service_pool.cc:225] Handling call
0623 08:04:35.402321 (+ 17519us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fb5a6 00007f19c7646616 00007f19c7646345 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:35.402574 (+ 253us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:35.402795 (+ 221us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:35.402940 (+ 145us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:35.403014 (+ 74us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:35.501867 (+ 98853us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:36.001239 (+499372us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.405055 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.405136 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.405140 (+ 4us) write_op.cc:435] Acquired schema lock
0623 08:04:35.405142 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:35.405219 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.405226 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.405231 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:35.405273 (+ 42us) tablet.cc:699] Row locks acquired
0623 08:04:35.405275 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:35.405303 (+ 28us) write_op.cc:270] Start()
0623 08:04:35.405328 (+ 25us) write_op.cc:276] Timestamp: P: 1750665875405272 usec, L: 0
0623 08:04:35.405333 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.405478 (+ 145us) log.cc:844] Serialized 405 byte log entry
0623 08:04:35.501255 (+ 95777us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.501360 (+ 105us) write_op.cc:301] APPLY: starting
0623 08:04:35.501375 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.501412 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.501446 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.501492 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.501522 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:35.501529 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.501599 (+ 70us) log.cc:844] Serialized 23 byte log entry
0623 08:04:35.501678 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.501693 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:35.501699 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.501768 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.501771 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":39689088,"child_traces":[["op",{"apply.queue_time_us":39,"apply.run_cpu_time_us":401,"apply.run_wall_time_us":401,"num_ops":1,"prepare.queue_time_us":1640,"prepare.run_cpu_time_us":473,"prepare.run_wall_time_us":472,"replication_time_us":95881}]]}
I20250623 08:04:36.002393 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 100) took 4297 ms. Trace:
I20250623 08:04:36.002395 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_739315d3797c465ca4ac459ddc3477e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:36.002471 979 rpcz_store.cc:276] 0623 08:04:31.704578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.704642 (+ 64us) service_pool.cc:225] Handling call
0623 08:04:36.002214 (+4297572us) ts_tablet_manager.cc:1145] Deleting tablet 6926c49b787941d98c647543bd3b9ed5
0623 08:04:36.002225 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.002335 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4297345}
I20250623 08:04:35.995152 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet d19d468ce13d4108a9badc36fc503306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:35.996577 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 18ea63263b9b466e922169eecafab747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:35542
W20250623 08:04:36.003033 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_739315d3797c465ca4ac459ddc3477e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.003062 981 log.cc:1199] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/67c44697ecab4dd9b43f544a602897ca
I20250623 08:04:36.003561 981 ts_tablet_manager.cc:1939] T 67c44697ecab4dd9b43f544a602897ca P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:36.004263 32057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23928d64bcef4c06af81698e688b7f94 consensus state has no leader
I20250623 08:04:36.004362 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 90) took 3576 ms. Trace:
I20250623 08:04:36.004477 981 rpcz_store.cc:276] 0623 08:04:32.428019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.428055 (+ 36us) service_pool.cc:225] Handling call
0623 08:04:35.875034 (+3446979us) ts_tablet_manager.cc:1145] Deleting tablet 67c44697ecab4dd9b43f544a602897ca
0623 08:04:35.875047 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:35.923510 (+ 48463us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:35.990165 (+ 66655us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.004326 (+ 14161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3446845}
I20250623 08:04:36.004652 32057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac" } modify_external_catalogs: true
I20250623 08:04:36.004822 32057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac" } modify_external_catalogs: true
I20250623 08:04:36.004860 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 91) took 3576 ms. Trace:
I20250623 08:04:36.004951 981 rpcz_store.cc:276] 0623 08:04:32.428284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.428320 (+ 36us) service_pool.cc:225] Handling call
0623 08:04:36.004731 (+3576411us) ts_tablet_manager.cc:1145] Deleting tablet 4fe30079045d437aa61bb5bfbfb96133
0623 08:04:36.004740 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.004827 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3576256}
I20250623 08:04:36.005317 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 92) took 3576 ms. Trace:
I20250623 08:04:36.005437 981 rpcz_store.cc:276] 0623 08:04:32.428623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.428662 (+ 39us) service_pool.cc:225] Handling call
0623 08:04:36.005194 (+3576532us) ts_tablet_manager.cc:1145] Deleting tablet 162308da898d4dde8ab5b1ab01d0f393
0623 08:04:36.005204 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.005288 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3576381}
I20250623 08:04:36.006278 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 101) took 4300 ms. Trace:
I20250623 08:04:36.006384 979 rpcz_store.cc:276] 0623 08:04:31.705329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.705390 (+ 61us) service_pool.cc:225] Handling call
0623 08:04:36.006113 (+4300723us) ts_tablet_manager.cc:1145] Deleting tablet 52d0a356c74f4daea9093c998b4bf73e
0623 08:04:36.006127 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.006251 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4300470}
I20250623 08:04:35.979120 32453 leader_election.cc:304] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:36.007681 979 tablet_replica.cc:331] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:36.007880 979 raft_consensus.cc:2241] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.007886 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet d19d468ce13d4108a9badc36fc503306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.008035 979 raft_consensus.cc:2270] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.008212 1359 raft_consensus.cc:2802] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:36.008337 1359 raft_consensus.cc:695] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:36.008602 1359 consensus_queue.cc:237] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.009450 848 raft_consensus.cc:491] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:36.009546 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 18ea63263b9b466e922169eecafab747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.009586 848 raft_consensus.cc:513] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.010217 848 leader_election.cc:290] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.005692 981 tablet_replica.cc:331] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.015251 981 raft_consensus.cc:2241] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.015697 981 raft_consensus.cc:2270] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.016269 981 ts_tablet_manager.cc:1905] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.019682 343 tablet_bootstrap.cc:492] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:36.019886 343 ts_tablet_manager.cc:1397] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.013s
I20250623 08:04:36.020581 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 776b412b3f564a7581c6a5e1c84e8499 (table default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652]) successfully deleted
I20250623 08:04:36.020733 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet 26f4a1b472e848ccb9a1ec71c1dd3516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.020730 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:36.020889 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 6926c49b787941d98c647543bd3b9ed5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6926c49b787941d98c647543bd3b9ed5 already in progress: creating tablet
W20250623 08:04:36.021098 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 52d0a356c74f4daea9093c998b4bf73e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52d0a356c74f4daea9093c998b4bf73e already in progress: creating tablet
I20250623 08:04:36.021340 343 raft_consensus.cc:357] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.021528 343 raft_consensus.cc:383] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.021595 343 raft_consensus.cc:738] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.021785 343 consensus_queue.cc:260] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.023453 343 ts_tablet_manager.cc:1428] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:36.023728 343 tablet_bootstrap.cc:492] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:36.023605 342 tablet_bootstrap.cc:492] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:35.997689 32056 rpcz_store.cc:276] 0623 08:04:34.623477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.488388 (+864911us) service_pool.cc:225] Handling call
0623 08:04:35.514733 (+ 26345us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fa306 00007f19c7644956 00007f19c7644685 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:35.515152 (+ 419us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:35.515319 (+ 167us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:35.515414 (+ 95us) catalog_manager.cc:2799] Locking tablets
0623 08:04:35.571850 (+ 56436us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:35.620914 (+ 49064us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:35.620990 (+ 76us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:35.997067 (+376077us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:35.997540 (+ 473us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.604740 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.604836 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.604844 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:35.604848 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:35.604946 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.604955 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.604960 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:35.605054 (+ 94us) tablet.cc:699] Row locks acquired
0623 08:04:35.605056 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:35.605087 (+ 31us) write_op.cc:270] Start()
0623 08:04:35.605114 (+ 27us) write_op.cc:276] Timestamp: P: 1750665875605034 usec, L: 0
0623 08:04:35.605117 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.605258 (+ 141us) log.cc:844] Serialized 725 byte log entry
0623 08:04:35.618598 (+ 13340us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.619273 (+ 675us) write_op.cc:301] APPLY: starting
0623 08:04:35.619286 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.619346 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.619395 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.619462 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.619502 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:35.619513 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:35.619578 (+ 65us) log.cc:844] Serialized 29 byte log entry
0623 08:04:35.619622 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.619640 (+ 18us) write_op.cc:454] Released schema lock
0623 08:04:35.619647 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.619725 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.619731 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":59875072,"child_traces":[["op",{"apply.queue_time_us":643,"apply.run_cpu_time_us":438,"apply.run_wall_time_us":438,"num_ops":2,"prepare.queue_time_us":1708,"prepare.run_cpu_time_us":588,"prepare.run_wall_time_us":588,"replication_time_us":13448}]]}
I20250623 08:04:36.025359 343 tablet_bootstrap.cc:654] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.025403 342 ts_tablet_manager.cc:1397] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.085s user 0.002s sys 0.002s
I20250623 08:04:36.026824 1417 raft_consensus.cc:491] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:36.026484 979 ts_tablet_manager.cc:1905] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.026999 1417 raft_consensus.cc:513] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:36.027622 1417 leader_election.cc:290] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:36.029560 952 ts_tablet_manager.cc:1918] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:35.996805 32058 rpcz_store.cc:276] 0623 08:04:34.704490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.507052 (+802562us) service_pool.cc:225] Handling call
0623 08:04:35.507458 (+ 406us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:35.507759 (+ 301us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:35.508003 (+ 244us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:35.508102 (+ 99us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:35.567337 (+ 59235us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:35.996668 (+429331us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.526273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.526381 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.526388 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:35.526390 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:35.526486 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.526500 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.526505 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:35.526564 (+ 59us) tablet.cc:699] Row locks acquired
0623 08:04:35.526567 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:35.526600 (+ 33us) write_op.cc:270] Start()
0623 08:04:35.526639 (+ 39us) write_op.cc:276] Timestamp: P: 1750665875526556 usec, L: 0
0623 08:04:35.526649 (+ 10us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.526865 (+ 216us) log.cc:844] Serialized 405 byte log entry
0623 08:04:35.566363 (+ 39498us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.566655 (+ 292us) write_op.cc:301] APPLY: starting
0623 08:04:35.566674 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.566713 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.566741 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.566808 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.566847 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:35.566853 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:35.566932 (+ 79us) log.cc:844] Serialized 23 byte log entry
0623 08:04:35.567197 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.567212 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:35.567218 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.567281 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.567284 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":233,"apply.run_cpu_time_us":433,"apply.run_wall_time_us":625,"num_ops":1,"prepare.queue_time_us":17665,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":1254,"replication_time_us":39665}]]}
I20250623 08:04:36.031949 32058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_049497d4290c4bdfaf1e609b8b2ac9ac" } new_table_name: "default.Table_34bf849295714bdab1ec5c40496934af" modify_external_catalogs: true
I20250623 08:04:36.034667 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_08ce069e682d42acac14ed36e986d074"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.035316 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08ce069e682d42acac14ed36e986d074 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.035800 952 log.cc:1199] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9918b15e48144ef0b8f3aedc2135be59
I20250623 08:04:36.034644 342 raft_consensus.cc:357] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.036556 342 raft_consensus.cc:383] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.039471 952 ts_tablet_manager.cc:1939] T 9918b15e48144ef0b8f3aedc2135be59 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:36.040325 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.040539 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 67c44697ecab4dd9b43f544a602897ca (table default.Table_815c7abc913347a688cc1772704c6715 [id=bb77edad430a486c898882bb9cd7bba8]) successfully deleted
I20250623 08:04:36.040627 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 26f4a1b472e848ccb9a1ec71c1dd3516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:35542
W20250623 08:04:36.040962 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 162308da898d4dde8ab5b1ab01d0f393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 162308da898d4dde8ab5b1ab01d0f393 already in progress: creating tablet
I20250623 08:04:36.040284 342 raft_consensus.cc:738] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.042233 981 ts_tablet_manager.cc:1918] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.042565 981 log.cc:1199] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/726d80e05a344f27b5a289723cb5aed7
I20250623 08:04:36.042806 1417 raft_consensus.cc:491] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.042956 1417 raft_consensus.cc:513] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.043393 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 58) took 5073 ms. Trace:
I20250623 08:04:36.043475 952 rpcz_store.cc:276] 0623 08:04:30.969961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.970248 (+ 287us) service_pool.cc:225] Handling call
0623 08:04:35.980586 (+5010338us) ts_tablet_manager.cc:1145] Deleting tablet 9918b15e48144ef0b8f3aedc2135be59
0623 08:04:35.980597 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.005291 (+ 24694us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.029450 (+ 24159us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.043372 (+ 13922us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67,"tablet-delete.queue_time_us":5010116}
I20250623 08:04:36.043668 1417 leader_election.cc:290] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.043803 952 tablet_replica.cc:331] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:36.044144 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0117cbb09dd4067836266c9d6a32adf" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.044354 32377 raft_consensus.cc:2466] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:36.044656 981 ts_tablet_manager.cc:1939] T 726d80e05a344f27b5a289723cb5aed7 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.046808 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 93) took 3617 ms. Trace:
I20250623 08:04:36.046305 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 26f4a1b472e848ccb9a1ec71c1dd3516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.047307 952 raft_consensus.cc:2241] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.047098 981 rpcz_store.cc:276] 0623 08:04:32.428928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.428969 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:36.005636 (+3576667us) ts_tablet_manager.cc:1145] Deleting tablet 726d80e05a344f27b5a289723cb5aed7
0623 08:04:36.005644 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.034266 (+ 28622us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.041997 (+ 7731us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.046715 (+ 4718us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3576481}
I20250623 08:04:36.047886 981 tablet_replica.cc:331] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.048318 981 raft_consensus.cc:2241] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.048774 981 raft_consensus.cc:2270] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.049822 952 raft_consensus.cc:2270] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.050379 952 ts_tablet_manager.cc:1905] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.051074 981 ts_tablet_manager.cc:1905] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:36.052142 32314 leader_election.cc:343] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: ce1f3ef3690c4837b95b105733884f1f
I20250623 08:04:36.049474 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.045336 342 consensus_queue.cc:260] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:36.052518 32453 leader_election.cc:343] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: f0117cbb09dd4067836266c9d6a32adf
I20250623 08:04:36.056751 342 ts_tablet_manager.cc:1428] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
W20250623 08:04:36.055503 32314 leader_election.cc:343] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 79731106915d4cebb9dcc3c10386ffa9
W20250623 08:04:36.060346 32314 leader_election.cc:343] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: ce1f3ef3690c4837b95b105733884f1f
I20250623 08:04:36.062271 32314 leader_election.cc:304] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:36.063503 981 ts_tablet_manager.cc:1918] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:36.063849 1417 raft_consensus.cc:2747] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:36.064209 981 log.cc:1199] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/c0f87fd62f1545089d00757f8214593a
I20250623 08:04:36.065099 981 ts_tablet_manager.cc:1939] T c0f87fd62f1545089d00757f8214593a P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.065759 339 tablet_bootstrap.cc:492] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.066105 339 ts_tablet_manager.cc:1397] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.000s
W20250623 08:04:36.065572 32314 leader_election.cc:343] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 79731106915d4cebb9dcc3c10386ffa9
I20250623 08:04:36.067050 339 raft_consensus.cc:357] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.066663 32314 leader_election.cc:304] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:36.067286 339 raft_consensus.cc:383] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.067374 339 raft_consensus.cc:738] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.067659 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 94) took 3638 ms. Trace:
I20250623 08:04:36.068142 981 rpcz_store.cc:276] 0623 08:04:32.429270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.429307 (+ 37us) service_pool.cc:225] Handling call
0623 08:04:36.047737 (+3618430us) ts_tablet_manager.cc:1145] Deleting tablet c0f87fd62f1545089d00757f8214593a
0623 08:04:36.047751 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.057519 (+ 9768us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.063283 (+ 5764us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.067575 (+ 4292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3618263}
I20250623 08:04:36.068866 981 tablet_replica.cc:331] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.069120 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 726d80e05a344f27b5a289723cb5aed7 (table default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]) successfully deleted
I20250623 08:04:36.069384 981 raft_consensus.cc:2241] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.069736 981 raft_consensus.cc:2270] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.070036 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet c0f87fd62f1545089d00757f8214593a (table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]) successfully deleted
I20250623 08:04:36.071128 981 ts_tablet_manager.cc:1905] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.067660 339 consensus_queue.cc:260] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.067981 1417 raft_consensus.cc:2747] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.071770 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 9918b15e48144ef0b8f3aedc2135be59 (table default.Table_c09ee28ff44c4c43835c2faa880a1c86 [id=371e56649fc6441c8aa52739de5aa077]) successfully deleted
I20250623 08:04:36.079516 343 tablet_bootstrap.cc:492] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:36.079672 343 ts_tablet_manager.cc:1397] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.004s
I20250623 08:04:36.080471 343 raft_consensus.cc:357] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.080591 1417 raft_consensus.cc:491] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.080655 343 raft_consensus.cc:383] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.080720 343 raft_consensus.cc:738] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.080760 1417 raft_consensus.cc:513] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.080922 343 consensus_queue.cc:260] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.081317 1417 leader_election.cc:290] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.082935 339 ts_tablet_manager.cc:1428] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250623 08:04:36.085913 339 tablet_bootstrap.cc:492] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.087184 343 ts_tablet_manager.cc:1428] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250623 08:04:36.087491 343 tablet_bootstrap.cc:492] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:36.086714 32453 leader_election.cc:304] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:36.091516 1359 raft_consensus.cc:2802] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:36.091715 1359 raft_consensus.cc:491] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:36.091863 1359 raft_consensus.cc:3058] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:36.092301 339 tablet_bootstrap.cc:654] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.093566 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7adec1482119405daca4aa603770ae22" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.095525 1359 raft_consensus.cc:513] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.096182 1359 leader_election.cc:290] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.096531 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7adec1482119405daca4aa603770ae22" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.097280 32314 leader_election.cc:304] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:36.097750 1417 raft_consensus.cc:2747] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.099552 342 tablet_bootstrap.cc:492] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:36.101217 342 tablet_bootstrap.cc:654] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.103386 981 ts_tablet_manager.cc:1918] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.103976 981 log.cc:1199] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/ce1f3ef3690c4837b95b105733884f1f
I20250623 08:04:36.106441 981 ts_tablet_manager.cc:1939] T ce1f3ef3690c4837b95b105733884f1f P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.108340 32377 raft_consensus.cc:3058] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.109423 952 ts_tablet_manager.cc:1918] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.110255 952 log.cc:1199] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/cde7df7d39a54114a4dcb28cfd1eb429
I20250623 08:04:36.110800 952 ts_tablet_manager.cc:1939] T cde7df7d39a54114a4dcb28cfd1eb429 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:36.111976 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 59) took 5141 ms. Trace:
I20250623 08:04:36.115360 952 rpcz_store.cc:276] 0623 08:04:30.970618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.970917 (+ 299us) service_pool.cc:225] Handling call
0623 08:04:36.043757 (+5072840us) ts_tablet_manager.cc:1145] Deleting tablet cde7df7d39a54114a4dcb28cfd1eb429
0623 08:04:36.043766 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.076044 (+ 32278us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.109320 (+ 33276us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.111958 (+ 2638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5072636}
I20250623 08:04:36.119021 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 60) took 5147 ms. Trace:
I20250623 08:04:36.119146 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 2f62ec83c88e43438b963e740d2c7004
I20250623 08:04:36.119941 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_9968742b8af441aa8db7bd5343fb4059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:36.120021 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_23e6e7311fd242248a3adec872f67c93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.120561 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9968742b8af441aa8db7bd5343fb4059 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:36.120617 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23e6e7311fd242248a3adec872f67c93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.123735 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f62ec83c88e43438b963e740d2c7004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.126731 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 95) took 3697 ms. Trace:
I20250623 08:04:36.126783 343 tablet_bootstrap.cc:654] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.127099 981 rpcz_store.cc:276] 0623 08:04:32.429552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.429586 (+ 34us) service_pool.cc:225] Handling call
0623 08:04:36.068727 (+3639141us) ts_tablet_manager.cc:1145] Deleting tablet ce1f3ef3690c4837b95b105733884f1f
0623 08:04:36.068740 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.094784 (+ 26044us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.102878 (+ 8094us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.126655 (+ 23777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3638925}
I20250623 08:04:36.128613 981 tablet_replica.cc:331] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.128144 339 tablet_bootstrap.cc:492] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.130677 339 ts_tablet_manager.cc:1397] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.000s
I20250623 08:04:36.131594 339 raft_consensus.cc:357] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.131845 339 raft_consensus.cc:383] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.131987 339 raft_consensus.cc:738] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.131989 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_7eaaf57b62b345609a21707908207acd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:36.132254 339 consensus_queue.cc:260] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:36.132737 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7eaaf57b62b345609a21707908207acd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.132923 339 ts_tablet_manager.cc:1428] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:36.135599 339 tablet_bootstrap.cc:492] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.136804 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_fdf4c673179d4d0d949c6c104c317a4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.137341 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdf4c673179d4d0d949c6c104c317a4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.132391 848 raft_consensus.cc:491] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 08:04:36.132635 32453 leader_election.cc:343] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: f0117cbb09dd4067836266c9d6a32adf
I20250623 08:04:36.139329 981 raft_consensus.cc:2241] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.139274 848 raft_consensus.cc:513] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.139652 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 776b412b3f564a7581c6a5e1c84e8499 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.139861 848 leader_election.cc:290] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.139986 952 rpcz_store.cc:276] 0623 08:04:30.971276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.975201 (+ 3925us) service_pool.cc:225] Handling call
0623 08:04:36.118917 (+5143716us) ts_tablet_manager.cc:1145] Deleting tablet ae7c2ac546114a819a543f1667fb7844
0623 08:04:36.118927 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.119005 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5143460}
I20250623 08:04:36.140166 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 with delete_type TABLET_DATA_DELETED (replaced by tablet 31a67b70cbc64bb3bfd5a34865109a01) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.140533 952 tablet_replica.cc:331] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:36.140733 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f with delete_type TABLET_DATA_DELETED (replaced by tablet b1aa596950da49a898e3ed9bbad8c45e) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.140865 981 raft_consensus.cc:2270] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.141191 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ea81642050141a9be84f151a5785abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.141361 981 ts_tablet_manager.cc:1905] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.141696 342 tablet_bootstrap.cc:492] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:36.141883 342 ts_tablet_manager.cc:1397] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.004s
I20250623 08:04:36.141943 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 162308da898d4dde8ab5b1ab01d0f393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.142264 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 205da880685b49b3b8669f73dc2c563b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:36.142549 32050 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:39129 (request call id 172) took 861 ms (client timeout 1000 ms). Trace:
I20250623 08:04:36.142702 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 9918b15e48144ef0b8f3aedc2135be59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.143003 342 raft_consensus.cc:357] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.143115 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f62ec83c88e43438b963e740d2c7004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.143754 32620 tablet_service.cc:1515] Processing DeleteTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 with delete_type TABLET_DATA_DELETED (replaced by tablet db81250d1e164a7d94affd764fc0ff0c) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.143990 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet a414144276f6484ea4690204752e2142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.144447 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet 38ecd4c243b840f4b66d8e01078dca64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.144624 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9e53da264a5ca1f70e1846c906f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.140679 339 tablet_bootstrap.cc:654] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.145165 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c6f98e2db1a4b0789c5819b2c7d4bb6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.145330 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet cde7df7d39a54114a4dcb28cfd1eb429 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]) successfully deleted
I20250623 08:04:36.145512 32656 raft_consensus.cc:2466] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:36.146304 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet ae7c2ac546114a819a543f1667fb7844 because the tablet was not found. No further retry: Not found: Tablet not found: ae7c2ac546114a819a543f1667fb7844
I20250623 08:04:36.147037 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_fea0e0f4beec4bddb07a0d24efe95aa4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:36.148144 1359 raft_consensus.cc:491] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.148293 1359 raft_consensus.cc:513] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.148950 1359 leader_election.cc:290] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.149902 32453 leader_election.cc:304] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:36.150841 1359 raft_consensus.cc:2802] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:36.150962 1359 raft_consensus.cc:491] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:36.151091 1359 raft_consensus.cc:3058] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.152952 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_d77ebc4a1078459db9cef97251f739ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:36.153277 981 ts_tablet_manager.cc:1918] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250623 08:04:36.153429 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d77ebc4a1078459db9cef97251f739ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.154043 1359 raft_consensus.cc:513] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.156016 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0117cbb09dd4067836266c9d6a32adf" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:36.156208 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c6f98e2db1a4b0789c5819b2c7d4bb6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.156323 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f62ec83c88e43438b963e740d2c7004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.156214 32376 raft_consensus.cc:3058] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:36.156592 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet ce1f3ef3690c4837b95b105733884f1f (table default.Table_5e8e88b2237f435595d619af47384b4e [id=dec4e6cfe22942698b2f2ae716f72e92]) successfully deleted
I20250623 08:04:36.150633 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_315a89d642a74362b8a04f8b82edd22f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.157177 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fea0e0f4beec4bddb07a0d24efe95aa4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.158810 342 raft_consensus.cc:383] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.159006 342 raft_consensus.cc:738] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.159338 342 consensus_queue.cc:260] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.159675 32376 raft_consensus.cc:2466] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 2.
I20250623 08:04:36.160255 342 ts_tablet_manager.cc:1428] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20250623 08:04:36.160528 342 tablet_bootstrap.cc:492] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:36.161674 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_315a89d642a74362b8a04f8b82edd22f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:36.142665 32050 rpcz_store.cc:269] 0623 08:04:35.280712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.324640 (+ 43928us) service_pool.cc:225] Handling call
0623 08:04:35.390107 (+ 65467us) spinlock_profiling.cc:243] Waited 57.6 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f3b58 00007f19c7640176 00007f19c763fea5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:36.142493 (+752386us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.601539 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.601665 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.601673 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:35.601676 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:35.601781 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.601797 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.601803 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:35.601886 (+ 83us) tablet.cc:699] Row locks acquired
0623 08:04:35.601889 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:35.601928 (+ 39us) write_op.cc:270] Start()
0623 08:04:35.601966 (+ 38us) write_op.cc:276] Timestamp: P: 1750665875601872 usec, L: 0
0623 08:04:35.601969 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.602136 (+ 167us) log.cc:844] Serialized 527 byte log entry
0623 08:04:35.618519 (+ 16383us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.618823 (+ 304us) write_op.cc:301] APPLY: starting
0623 08:04:35.618838 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.618880 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.618914 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.618960 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.619005 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:35.619013 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:35.619079 (+ 66us) log.cc:844] Serialized 23 byte log entry
0623 08:04:35.619129 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.619144 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:35.619150 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.619210 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.619213 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":53,"spinlock_wait_cycles":132428544,"child_traces":[["op",{"apply.queue_time_us":257,"apply.run_cpu_time_us":382,"apply.run_wall_time_us":382,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":663,"prepare.run_wall_time_us":663,"replication_time_us":16517}]]}
I20250623 08:04:36.163236 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b19e5f2359a84c7ea02a0b25240a54cb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.163511 32656 raft_consensus.cc:2466] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:36.165633 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b19e5f2359a84c7ea02a0b25240a54cb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.166448 342 tablet_bootstrap.cc:654] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:36.167452 32453 consensus_peers.cc:487] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:36.167769 32453 leader_election.cc:343] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.169310 32453 leader_election.cc:304] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
W20250623 08:04:36.169456 32058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fd478c007914d2cb09d929b50322a64 consensus state has no leader
I20250623 08:04:36.169776 848 raft_consensus.cc:2802] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:36.169903 848 raft_consensus.cc:695] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:36.170159 32057 master_service.cc:948] ReplaceTablet: received request to replace tablet d6ed4cb252b1452087f8ee53ec78d985 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:36.170071 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_9c574fd66ea84e8eb1cae99bbb828288"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:36.170143 848 consensus_queue.cc:237] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:36.171402 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c574fd66ea84e8eb1cae99bbb828288 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.173954 32050 master_service.cc:948] ReplaceTablet: received request to replace tablet 9bfa787f828b411ab68d81bb896f7fb6 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:36.176754 32058 master_service.cc:948] ReplaceTablet: received request to replace tablet b84b8b8713774ab4896842feb9fb075e from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:36.176776 32028 master_runner.cc:186] Error getting master registration for 127.31.69.61:33313: OK, instance_id { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" instance_seqno: 1750665874351111 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250623 08:04:36.177165 32028 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250623 08:04:36.179450 1359 leader_election.cc:290] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.164562 979 ts_tablet_manager.cc:1918] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.180007 979 log.cc:1199] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/a9c8d480b8ba49a1be140b0e0a79f4e4
I20250623 08:04:36.169184 343 tablet_bootstrap.cc:492] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:36.180505 343 ts_tablet_manager.cc:1397] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.002s
I20250623 08:04:36.180950 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 16331ba6a34942a5a1b4ff4ac7bac55e
I20250623 08:04:36.181543 343 raft_consensus.cc:357] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.181773 343 raft_consensus.cc:383] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.181933 343 raft_consensus.cc:738] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.182317 343 consensus_queue.cc:260] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.182708 32054 catalog_manager.cc:6652] ReplaceTablet: tablet 16331ba6a34942a5a1b4ff4ac7bac55e deleted and replaced by tablet 4e3920ac85e54379abbcc42dd80db504
I20250623 08:04:36.182847 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 16331ba6a34942a5a1b4ff4ac7bac55e with delete_type TABLET_DATA_DELETED (replaced by tablet 4e3920ac85e54379abbcc42dd80db504) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.183187 343 ts_tablet_manager.cc:1428] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:36.185012 981 log.cc:1199] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/4ea81642050141a9be84f151a5785abc
I20250623 08:04:36.185577 981 ts_tablet_manager.cc:1939] T 4ea81642050141a9be84f151a5785abc P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.186383 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 16331ba6a34942a5a1b4ff4ac7bac55e with delete_type TABLET_DATA_DELETED (replaced by tablet 4e3920ac85e54379abbcc42dd80db504) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.186987 1417 raft_consensus.cc:491] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.187161 979 ts_tablet_manager.cc:1939] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:36.187144 1417 raft_consensus.cc:513] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.187749 1417 leader_election.cc:290] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.187884 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 104) took 4275 ms. Trace:
I20250623 08:04:36.187980 979 rpcz_store.cc:276] 0623 08:04:31.912590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.912665 (+ 75us) service_pool.cc:225] Handling call
0623 08:04:36.007638 (+4094973us) ts_tablet_manager.cc:1145] Deleting tablet a9c8d480b8ba49a1be140b0e0a79f4e4
0623 08:04:36.007647 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.129666 (+122019us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.164402 (+ 34736us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.187860 (+ 23458us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4094666}
I20250623 08:04:36.188369 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 96) took 3758 ms. Trace:
I20250623 08:04:36.140739 952 raft_consensus.cc:2241] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.188720 981 rpcz_store.cc:276] 0623 08:04:32.429891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.429929 (+ 38us) service_pool.cc:225] Handling call
0623 08:04:36.128529 (+3698600us) ts_tablet_manager.cc:1145] Deleting tablet 4ea81642050141a9be84f151a5785abc
0623 08:04:36.128539 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.149121 (+ 20582us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.153155 (+ 4034us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.188109 (+ 34954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3698446}
I20250623 08:04:36.189049 952 raft_consensus.cc:2270] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.189126 981 tablet_replica.cc:331] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.192124 952 ts_tablet_manager.cc:1905] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.193332 981 raft_consensus.cc:2241] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.193593 981 raft_consensus.cc:2270] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.194311 343 tablet_bootstrap.cc:492] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:36.194906 342 tablet_bootstrap.cc:492] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:36.195067 342 ts_tablet_manager.cc:1397] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250623 08:04:36.195209 339 tablet_bootstrap.cc:492] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.195456 339 ts_tablet_manager.cc:1397] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.006s
I20250623 08:04:36.195863 342 raft_consensus.cc:357] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.196166 342 raft_consensus.cc:383] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.198096 339 raft_consensus.cc:357] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.198280 339 raft_consensus.cc:383] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.196231 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "965f1f0a107b4653b6aac01e0d8e259a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.198362 339 raft_consensus.cc:738] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.198449 32513 raft_consensus.cc:2466] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:36.198640 339 consensus_queue.cc:260] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.198894 342 raft_consensus.cc:738] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.199260 339 ts_tablet_manager.cc:1428] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250623 08:04:36.199259 342 consensus_queue.cc:260] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.199575 339 tablet_bootstrap.cc:492] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.199906 342 ts_tablet_manager.cc:1428] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:36.200201 342 tablet_bootstrap.cc:492] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:36.201328 339 tablet_bootstrap.cc:654] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.201838 342 tablet_bootstrap.cc:654] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.202482 981 ts_tablet_manager.cc:1905] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.202951 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet a9c8d480b8ba49a1be140b0e0a79f4e4 (table default.Table_4550e74cc6e845719c1466d3acd1b723 [id=587d6896c7dd40838ce8ad01026379a8]) successfully deleted
I20250623 08:04:36.203593 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c6f98e2db1a4b0789c5819b2c7d4bb6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:36.203616 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c6f98e2db1a4b0789c5819b2c7d4bb6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:36.203781 32646 raft_consensus.cc:3058] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.206478 32646 raft_consensus.cc:2466] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:36.203662 32314 leader_election.cc:304] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:36.207289 1332 raft_consensus.cc:2802] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:36.207394 1332 raft_consensus.cc:491] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:36.207492 1332 raft_consensus.cc:3058] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.209290 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 4ea81642050141a9be84f151a5785abc (table default.Table_176ee9ee2de7444ca3c54ee24e3e5667 [id=1475deb3ccfa49d5935900afa29dee6f]) successfully deleted
I20250623 08:04:36.209650 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c382db2a0c144ec4abd51b3e927e0a17" } modify_external_catalogs: true
I20250623 08:04:36.211867 32630 tablet_service.cc:1468] Processing CreateTablet for tablet b720fdea539f423ba2d2aa9f89b09ae9 (DEFAULT_TABLE table=default.Table_59f2123c149e43569f3e3db991d1d09c [id=107d7d6ec5da4d29a8dbd45ba77eb017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.212239 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b720fdea539f423ba2d2aa9f89b09ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.212358 1332 raft_consensus.cc:513] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.212788 32057 catalog_manager.cc:5582] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:36.212983 1332 leader_election.cc:290] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.213157 32051 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d77ebc4a1078459db9cef97251f739ff" } modify_external_catalogs: true
I20250623 08:04:36.213311 32051 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d77ebc4a1078459db9cef97251f739ff" } modify_external_catalogs: true
I20250623 08:04:36.215548 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet b84b8b8713774ab4896842feb9fb075e
I20250623 08:04:36.215997 32058 catalog_manager.cc:6652] ReplaceTablet: tablet b84b8b8713774ab4896842feb9fb075e deleted and replaced by tablet a3576624314d4e85b411aab9109683f7
I20250623 08:04:36.216441 32632 tablet_service.cc:1515] Processing DeleteTablet for tablet b84b8b8713774ab4896842feb9fb075e with delete_type TABLET_DATA_DELETED (replaced by tablet a3576624314d4e85b411aab9109683f7) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:36.217411 32314 leader_election.cc:343] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 965f1f0a107b4653b6aac01e0d8e259a
I20250623 08:04:36.219084 343 tablet_bootstrap.cc:654] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.223455 981 ts_tablet_manager.cc:1918] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.223452 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:36.223598 981 log.cc:1199] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/e22f9bae5f7248af99cec06ce6dbbd41
I20250623 08:04:36.223824 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:36.224020 981 ts_tablet_manager.cc:1939] T e22f9bae5f7248af99cec06ce6dbbd41 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.224937 32057 catalog_manager.cc:5582] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:36.225327 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:36.227927 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:36.228379 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:36.228600 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 16331ba6a34942a5a1b4ff4ac7bac55e with delete_type TABLET_DATA_DELETED (replaced by tablet 4e3920ac85e54379abbcc42dd80db504) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.228611 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 97) took 3794 ms. Trace:
I20250623 08:04:36.228710 981 rpcz_store.cc:276] 0623 08:04:32.430341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.430382 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:36.189067 (+3758685us) ts_tablet_manager.cc:1145] Deleting tablet e22f9bae5f7248af99cec06ce6dbbd41
0623 08:04:36.189075 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.206325 (+ 17250us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.223326 (+ 17001us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.224769 (+ 1443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3758505}
I20250623 08:04:36.229199 32050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 9bfa787f828b411ab68d81bb896f7fb6
I20250623 08:04:36.229159 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c382db2a0c144ec4abd51b3e927e0a17" } modify_external_catalogs: true
I20250623 08:04:36.229403 981 tablet_replica.cc:331] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.229605 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.229610 981 raft_consensus.cc:2241] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.229804 981 raft_consensus.cc:2270] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.229812 32058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3d6c2aca7a944301bc67f7901cfef8b3" } modify_external_catalogs: true
I20250623 08:04:36.229974 32058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3d6c2aca7a944301bc67f7901cfef8b3" } modify_external_catalogs: true
I20250623 08:04:36.230108 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet b84b8b8713774ab4896842feb9fb075e with delete_type TABLET_DATA_DELETED (replaced by tablet a3576624314d4e85b411aab9109683f7) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.230500 32050 catalog_manager.cc:6652] ReplaceTablet: tablet 9bfa787f828b411ab68d81bb896f7fb6 deleted and replaced by tablet 886fc0fa39c54bddb25ac01f68d95805
I20250623 08:04:36.230791 32490 tablet_service.cc:1468] Processing CreateTablet for tablet b720fdea539f423ba2d2aa9f89b09ae9 (DEFAULT_TABLE table=default.Table_59f2123c149e43569f3e3db991d1d09c [id=107d7d6ec5da4d29a8dbd45ba77eb017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.230950 32055 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fdf4c673179d4d0d949c6c104c317a4a" } new_table_name: "default.Table_a8ada1b74f854fe58c88fc3d90167bd3" modify_external_catalogs: true
I20250623 08:04:36.231065 981 ts_tablet_manager.cc:1905] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.231164 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b720fdea539f423ba2d2aa9f89b09ae9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:36.235078 32059 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:39501 (request call id 168) took 1381 ms (client timeout 1000 ms). Trace:
I20250623 08:04:36.235934 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "965f1f0a107b4653b6aac01e0d8e259a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:36.236132 32518 raft_consensus.cc:3058] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.236267 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_78fab2c6cc5c4523823de72525524e88" } new_table_name: "default.Table_4248729f2b014a7cbe5da578d4a1e5af" modify_external_catalogs: true
I20250623 08:04:36.237392 32053 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_57b13d90b3894f80b5a1cec576d2fb3a" } modify_external_catalogs: true
I20250623 08:04:36.238691 32518 raft_consensus.cc:2466] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:36.246055 342 tablet_bootstrap.cc:492] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:36.246378 342 ts_tablet_manager.cc:1397] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20250623 08:04:36.247123 32453 leader_election.cc:304] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:36.247954 848 raft_consensus.cc:2802] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:36.248098 848 raft_consensus.cc:491] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:36.248255 848 raft_consensus.cc:3058] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.248641 981 ts_tablet_manager.cc:1918] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.250214 981 log.cc:1199] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/35ab9f77a81a4490abb0d2c0c69cd90f
I20250623 08:04:36.251215 981 ts_tablet_manager.cc:1939] T 35ab9f77a81a4490abb0d2c0c69cd90f P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.251428 32453 leader_election.cc:304] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:36.252585 1359 raft_consensus.cc:2802] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:36.252933 1359 raft_consensus.cc:695] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:36.253271 1359 consensus_queue.cc:237] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.254858 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 98) took 3824 ms. Trace:
I20250623 08:04:36.255908 848 raft_consensus.cc:513] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.256850 848 leader_election.cc:290] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.265864 1413 tablet.cc:1722] T 74842fde5152419ca35f3950e465247a P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:36.268677 32051 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 54bd648489c74a26971e8fc26fd5c807
I20250623 08:04:36.269059 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3fd60c781aaa4cbfa2e69d06f765bc0c" } new_table_name: "default.Table_83732a7d68bb412f9c26b799fc16e4ec" modify_external_catalogs: true
I20250623 08:04:36.227924 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 105) took 4301 ms. Trace:
I20250623 08:04:36.240885 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 886fc0fa39c54bddb25ac01f68d95805) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.252336 32632 tablet_service.cc:1468] Processing CreateTablet for tablet f5f58f55070344ff98a3808f70e0c755 (DEFAULT_TABLE table=default.Table_57b13d90b3894f80b5a1cec576d2fb3a [id=431858068ed8423e91969be835dc8378]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.270116 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f58f55070344ff98a3808f70e0c755. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.255357 981 rpcz_store.cc:276] 0623 08:04:32.430684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.430720 (+ 36us) service_pool.cc:225] Handling call
0623 08:04:36.229311 (+3798591us) ts_tablet_manager.cc:1145] Deleting tablet 35ab9f77a81a4490abb0d2c0c69cd90f
0623 08:04:36.229320 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.234155 (+ 4835us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.248461 (+ 14306us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.254814 (+ 6353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3798392}
I20250623 08:04:36.270624 981 tablet_replica.cc:331] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: stopping tablet replica
W20250623 08:04:36.270807 32055 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 297bdf6c0e424190a35b55cbeefe7161 has no consensus state
I20250623 08:04:36.270879 981 raft_consensus.cc:2241] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.271116 981 pending_rounds.cc:70] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:36.271186 981 pending_rounds.cc:77] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7170727425326264320 op_type: NO_OP noop_request { }
I20250623 08:04:36.271391 32053 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_57b13d90b3894f80b5a1cec576d2fb3a" } modify_external_catalogs: true
I20250623 08:04:36.271431 342 raft_consensus.cc:357] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.271915 342 raft_consensus.cc:383] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.272116 342 raft_consensus.cc:738] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.252293 32631 tablet_service.cc:1468] Processing CreateTablet for tablet a383d10d53b04d5699dd3c4bad8878c7 (DEFAULT_TABLE table=default.Table_68d8428833794f57b663cdae5b7d6cb3 [id=bcf8b85a65544390b35245e6ce6f7336]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:36.273104 32453 leader_election.cc:343] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.252696 32628 tablet_service.cc:1468] Processing CreateTablet for tablet c7f759b707ef46df99873b92c694ad6b (DEFAULT_TABLE table=default.Table_31ab57df769c4b2cb4e2cb74d69b4eb5 [id=92602c2df9c44292abbed7d6095922c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.273253 981 raft_consensus.cc:2887] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:36.273407 981 raft_consensus.cc:2270] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.273530 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f759b707ef46df99873b92c694ad6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:36.274063 32453 leader_election.cc:343] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.275122 981 ts_tablet_manager.cc:1905] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.272590 342 consensus_queue.cc:260] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.253057 32620 tablet_service.cc:1468] Processing CreateTablet for tablet baccec88aefb4056be9578bca13f1591 (DEFAULT_TABLE table=default.Table_5afa01a4e2fa4b7c97e51a681da648d0 [id=0a1bbca429e6427581026c4600755351]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.279028 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baccec88aefb4056be9578bca13f1591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.279853 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b19e5f2359a84c7ea02a0b25240a54cb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:36.280122 32646 raft_consensus.cc:3058] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.282951 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a383d10d53b04d5699dd3c4bad8878c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.283762 32646 raft_consensus.cc:2466] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:36.286059 1426 tablet.cc:1722] T 74842fde5152419ca35f3950e465247a P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:36.263325 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 0cca63991d5b4ef5bd8a9181d127ebe3 (DEFAULT_TABLE table=default.Table_da00c692e65345c496f4e52c106aa2e6 [id=5ea293756d514f06b8d749c615417969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.286880 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cca63991d5b4ef5bd8a9181d127ebe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.264495 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 8911308855cb4414be426dbcec592f4a (DEFAULT_TABLE table=default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.290132 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8911308855cb4414be426dbcec592f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.266692 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 4e06ef06db2643779f93617e27f100dd (DEFAULT_TABLE table=default.Table_8a32f131cc9048a583663a64ed26a4af [id=e201b6e61b4641b9b4894576ac00fad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.291664 1303 raft_consensus.cc:491] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.291841 1303 raft_consensus.cc:513] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.292483 1303 leader_election.cc:290] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.294003 1359 raft_consensus.cc:491] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.294143 1359 raft_consensus.cc:513] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.295933 1359 leader_election.cc:290] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.297724 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 886fc0fa39c54bddb25ac01f68d95805) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.298421 32490 tablet_service.cc:1468] Processing CreateTablet for tablet f5f58f55070344ff98a3808f70e0c755 (DEFAULT_TABLE table=default.Table_57b13d90b3894f80b5a1cec576d2fb3a [id=431858068ed8423e91969be835dc8378]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.298856 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f58f55070344ff98a3808f70e0c755. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.302130 1440 raft_consensus.cc:491] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.302271 1440 raft_consensus.cc:513] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.302340 32055 master_service.cc:948] ReplaceTablet: received request to replace tablet 771ecfc41e2b4007a4d213cec5b128c6 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:36.302922 1440 leader_election.cc:290] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:36.304175 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e602d391526d4b0eb0dbf5189afbe075" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:36.304700 32589 leader_election.cc:343] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.308604 342 ts_tablet_manager.cc:1428] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.062s user 0.002s sys 0.001s
I20250623 08:04:36.282703 981 ts_tablet_manager.cc:1918] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250623 08:04:36.309724 981 log.cc:1199] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/9bc95f12ab5d4b9b9f0729cb88aabbaa
I20250623 08:04:36.310222 981 ts_tablet_manager.cc:1939] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.310685 342 tablet_bootstrap.cc:492] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:36.311086 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 99) took 3880 ms. Trace:
I20250623 08:04:36.311254 981 rpcz_store.cc:276] 0623 08:04:32.431149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.431290 (+ 141us) service_pool.cc:225] Handling call
0623 08:04:36.270548 (+3839258us) ts_tablet_manager.cc:1145] Deleting tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa
0623 08:04:36.270561 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.279572 (+ 9011us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.282584 (+ 3012us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.311050 (+ 28466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3839054}
I20250623 08:04:36.311638 981 tablet_replica.cc:331] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.311880 981 raft_consensus.cc:2241] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.312132 981 raft_consensus.cc:2270] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.312538 32490 tablet_service.cc:1468] Processing CreateTablet for tablet a383d10d53b04d5699dd3c4bad8878c7 (DEFAULT_TABLE table=default.Table_68d8428833794f57b663cdae5b7d6cb3 [id=bcf8b85a65544390b35245e6ce6f7336]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.312795 981 ts_tablet_manager.cc:1905] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.313104 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a383d10d53b04d5699dd3c4bad8878c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.319778 981 ts_tablet_manager.cc:1918] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.313632 339 tablet_bootstrap.cc:492] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.320034 339 ts_tablet_manager.cc:1397] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.121s user 0.014s sys 0.013s
I20250623 08:04:36.320466 848 raft_consensus.cc:491] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7351aa31800245659edc9837e27daffd)
I20250623 08:04:36.320633 848 raft_consensus.cc:513] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:36.320842 339 raft_consensus.cc:357] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.320997 339 raft_consensus.cc:383] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.321070 339 raft_consensus.cc:738] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.321287 339 consensus_queue.cc:260] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.321372 848 leader_election.cc:290] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:36.321779 339 ts_tablet_manager.cc:1428] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:36.322053 339 tablet_bootstrap.cc:492] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.323014 342 tablet_bootstrap.cc:654] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.323657 339 tablet_bootstrap.cc:654] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.324606 32491 tablet_service.cc:1468] Processing CreateTablet for tablet baccec88aefb4056be9578bca13f1591 (DEFAULT_TABLE table=default.Table_5afa01a4e2fa4b7c97e51a681da648d0 [id=0a1bbca429e6427581026c4600755351]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.325089 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baccec88aefb4056be9578bca13f1591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.325508 32492 tablet_service.cc:1468] Processing CreateTablet for tablet c7f759b707ef46df99873b92c694ad6b (DEFAULT_TABLE table=default.Table_31ab57df769c4b2cb4e2cb74d69b4eb5 [id=92602c2df9c44292abbed7d6095922c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.325883 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f759b707ef46df99873b92c694ad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.329371 981 log.cc:1199] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/776b412b3f564a7581c6a5e1c84e8499
I20250623 08:04:36.329996 981 ts_tablet_manager.cc:1939] T 776b412b3f564a7581c6a5e1c84e8499 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.331557 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 100) took 3900 ms. Trace:
I20250623 08:04:36.332252 981 rpcz_store.cc:276] 0623 08:04:32.431257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.431517 (+ 260us) service_pool.cc:225] Handling call
0623 08:04:36.311580 (+3880063us) ts_tablet_manager.cc:1145] Deleting tablet 776b412b3f564a7581c6a5e1c84e8499
0623 08:04:36.311590 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.316683 (+ 5093us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.319665 (+ 2982us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.331500 (+ 11835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3879889}
I20250623 08:04:36.332932 981 tablet_replica.cc:331] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.333410 981 raft_consensus.cc:2241] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.333508 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f5f58f55070344ff98a3808f70e0c755
I20250623 08:04:36.333976 981 raft_consensus.cc:2270] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.334441 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_a50378b31cda420a8791443844eed5f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.335147 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a50378b31cda420a8791443844eed5f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.335173 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 8911308855cb4414be426dbcec592f4a (DEFAULT_TABLE table=default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.335477 981 ts_tablet_manager.cc:1905] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.335600 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8911308855cb4414be426dbcec592f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.338786 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 0cca63991d5b4ef5bd8a9181d127ebe3 (DEFAULT_TABLE table=default.Table_da00c692e65345c496f4e52c106aa2e6 [id=5ea293756d514f06b8d749c615417969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.339102 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cca63991d5b4ef5bd8a9181d127ebe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.339490 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 4e06ef06db2643779f93617e27f100dd (DEFAULT_TABLE table=default.Table_8a32f131cc9048a583663a64ed26a4af [id=e201b6e61b4641b9b4894576ac00fad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.339865 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e06ef06db2643779f93617e27f100dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.340979 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet f5f58f55070344ff98a3808f70e0c755 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.341677 979 rpcz_store.cc:276] 0623 08:04:31.926734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.927020 (+ 286us) service_pool.cc:225] Handling call
0623 08:04:36.207833 (+4280813us) ts_tablet_manager.cc:1145] Deleting tablet f3c850021e914530bb106ab7166be7c9
0623 08:04:36.207845 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.227905 (+ 20060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4260896}
I20250623 08:04:36.342478 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 106) took 4415 ms. Trace:
I20250623 08:04:36.342557 979 rpcz_store.cc:276] 0623 08:04:31.926953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.927748 (+ 795us) service_pool.cc:225] Handling call
0623 08:04:36.342199 (+4414451us) ts_tablet_manager.cc:1145] Deleting tablet 1e47b10f46b8402e94354ebca573c194
0623 08:04:36.342208 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.342462 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4414222}
W20250623 08:04:36.342994 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f3c850021e914530bb106ab7166be7c9 because the tablet was not found. No further retry: Not found: Tablet not found: f3c850021e914530bb106ab7166be7c9
I20250623 08:04:36.343263 339 tablet_bootstrap.cc:492] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
W20250623 08:04:36.343256 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 1e47b10f46b8402e94354ebca573c194 because the tablet was not found. No further retry: Not found: Tablet not found: 1e47b10f46b8402e94354ebca573c194
I20250623 08:04:36.343429 339 ts_tablet_manager.cc:1397] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20250623 08:04:36.343570 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 107) took 4409 ms. Trace:
I20250623 08:04:36.343649 979 rpcz_store.cc:276] 0623 08:04:31.934325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.934995 (+ 670us) service_pool.cc:225] Handling call
0623 08:04:36.343477 (+4408482us) ts_tablet_manager.cc:1145] Deleting tablet 417a050eada4403ebbddff255fb6fb5a
0623 08:04:36.343487 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.343554 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4408268}
I20250623 08:04:36.344060 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 108) took 4409 ms. Trace:
I20250623 08:04:36.344141 979 rpcz_store.cc:276] 0623 08:04:31.934461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.935243 (+ 782us) service_pool.cc:225] Handling call
0623 08:04:36.343966 (+4408723us) ts_tablet_manager.cc:1145] Deleting tablet 175ee90133994154b7e4b58b0770db3b
0623 08:04:36.343974 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.344040 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4408544}
I20250623 08:04:36.344201 339 raft_consensus.cc:357] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.344445 339 raft_consensus.cc:383] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 08:04:36.344523 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 417a050eada4403ebbddff255fb6fb5a because the tablet was not found. No further retry: Not found: Tablet not found: 417a050eada4403ebbddff255fb6fb5a
I20250623 08:04:36.344569 339 raft_consensus.cc:738] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:36.344808 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 175ee90133994154b7e4b58b0770db3b because the tablet was not found. No further retry: Not found: Tablet not found: 175ee90133994154b7e4b58b0770db3b
I20250623 08:04:36.344919 339 consensus_queue.cc:260] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.345399 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 110) took 4410 ms. Trace:
I20250623 08:04:36.345479 979 rpcz_store.cc:276] 0623 08:04:31.934652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.935453 (+ 801us) service_pool.cc:225] Handling call
0623 08:04:36.345321 (+4409868us) ts_tablet_manager.cc:1145] Deleting tablet 7da84969a6464b5caeaf47af0cbaaa4d
0623 08:04:36.345330 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.345385 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4409688}
I20250623 08:04:36.345659 339 ts_tablet_manager.cc:1428] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20250623 08:04:36.345871 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 7da84969a6464b5caeaf47af0cbaaa4d because the tablet was not found. No further retry: Not found: Tablet not found: 7da84969a6464b5caeaf47af0cbaaa4d
I20250623 08:04:36.346051 339 tablet_bootstrap.cc:492] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.346176 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 111) took 4411 ms. Trace:
I20250623 08:04:36.346246 979 rpcz_store.cc:276] 0623 08:04:31.934794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.935662 (+ 868us) service_pool.cc:225] Handling call
0623 08:04:36.346096 (+4410434us) ts_tablet_manager.cc:1145] Deleting tablet 5629d11c44e241d598f3e519b2fe1a44
0623 08:04:36.346106 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.346162 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4410258}
W20250623 08:04:36.346901 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 5629d11c44e241d598f3e519b2fe1a44 because the tablet was not found. No further retry: Not found: Tablet not found: 5629d11c44e241d598f3e519b2fe1a44
W20250623 08:04:36.268620 32059 rpcz_store.cc:269] 0623 08:04:34.853149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.911681 (+ 58532us) service_pool.cc:225] Handling call
0623 08:04:36.235044 (+1323363us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:35.300186 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:35.300309 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:35.300316 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:35.300318 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:35.300519 (+ 201us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:35.300530 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:35.300535 (+ 5us) tablet.cc:683] Acquiring locks for 6 operations
0623 08:04:35.300710 (+ 175us) tablet.cc:699] Row locks acquired
0623 08:04:35.300713 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:35.300753 (+ 40us) write_op.cc:270] Start()
0623 08:04:35.300810 (+ 57us) write_op.cc:276] Timestamp: P: 1750665875300732 usec, L: 0
0623 08:04:35.300814 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:35.386023 (+ 85209us) log.cc:844] Serialized 2398 byte log entry
0623 08:04:35.386244 (+ 221us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 000000000097041a 00007f19c8ac25e2 00007f19c9e69255 00007f19c9e67d79 00007f19c9e701a0 00007f19c9e6ffad 0000000000a751bc 00007f19bd0d43ff 00007f19bd0d76cb 00007f19bd0d7548 0000000000a751bc
0623 08:04:35.406153 (+ 19909us) op_driver.cc:464] REPLICATION: finished
0623 08:04:35.406936 (+ 783us) write_op.cc:301] APPLY: starting
0623 08:04:35.406950 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:35.406995 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:35.406998 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:35.407098 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:35.407132 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0623 08:04:35.407139 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:35.407221 (+ 82us) log.cc:844] Serialized 53 byte log entry
0623 08:04:35.407285 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.407302 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:35.407309 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:35.407370 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:35.407375 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25841,"reported_tablets":174,"child_traces":[["op",{"apply.queue_time_us":726,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":431,"num_ops":6,"prepare.queue_time_us":2933,"prepare.run_cpu_time_us":2288,"prepare.run_wall_time_us":86127,"raft.queue_time_us":10027,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":306,"replication_time_us":105313,"spinlock_wait_cycles":195471360}]]}
I20250623 08:04:36.347990 32059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_4e377b569121400d82096403060f9cda" } modify_external_catalogs: true
I20250623 08:04:36.348182 32059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_4e377b569121400d82096403060f9cda" } modify_external_catalogs: true
I20250623 08:04:36.350085 952 ts_tablet_manager.cc:1918] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.350368 952 log.cc:1199] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9bc95f12ab5d4b9b9f0729cb88aabbaa
I20250623 08:04:36.351171 952 ts_tablet_manager.cc:1939] T 9bc95f12ab5d4b9b9f0729cb88aabbaa P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:36.352095 339 tablet_bootstrap.cc:654] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.353979 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:36.354084 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 89a24019b797467e857b744d701fb1ed (DEFAULT_TABLE table=default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.354346 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_78fab2c6cc5c4523823de72525524e88 [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 1 got 0
I20250623 08:04:36.354504 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a24019b797467e857b744d701fb1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.354578 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_32db9f68dd974fb8aff372a8eeccce3d [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 1 got 0
I20250623 08:04:36.354846 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]). Expected version 1 got 0
I20250623 08:04:36.355322 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 74c1cdc83ffd4a09a163479ac58fd405 (DEFAULT_TABLE table=default.Table_fbb1fc5910454112b795f72c26f1af5b [id=6d6ee37c928745ef877266573fe021f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.355636 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c1cdc83ffd4a09a163479ac58fd405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.355602 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:36.355849 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:36.356155 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:36.356415 981 ts_tablet_manager.cc:1918] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.357388 981 log.cc:1199] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/9918b15e48144ef0b8f3aedc2135be59
I20250623 08:04:36.359050 981 ts_tablet_manager.cc:1939] T 9918b15e48144ef0b8f3aedc2135be59 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.359148 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 185c2bf7b9254947bb76b7fe0924559e (DEFAULT_TABLE table=default.Table_06b731510ae743608f1fbc4d250acf14 [id=7fbe15151173464ea0ee480720c58b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.359735 32632 tablet_service.cc:1468] Processing CreateTablet for tablet e7b9e843ff1344a1aaacf038f15f64cd (DEFAULT_TABLE table=default.Table_d74a4572f91e48d0b1c123e11ecf5b51 [id=e1e321dbfa254d21b11fc31f07d4de07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.359949 32056 catalog_manager.cc:5582] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:36.360276 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 101) took 3928 ms. Trace:
I20250623 08:04:36.361264 981 rpcz_store.cc:276] 0623 08:04:32.431799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.431838 (+ 39us) service_pool.cc:225] Handling call
0623 08:04:36.332850 (+3901012us) ts_tablet_manager.cc:1145] Deleting tablet 9918b15e48144ef0b8f3aedc2135be59
0623 08:04:36.332859 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.349987 (+ 17128us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.356288 (+ 6301us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.360238 (+ 3950us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3900823}
I20250623 08:04:36.361935 981 tablet_replica.cc:331] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.361918 32453 leader_election.cc:304] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:36.362457 981 raft_consensus.cc:2241] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.362989 1359 consensus_queue.cc:1035] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:36.363514 1359 raft_consensus.cc:2802] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:36.363811 1359 raft_consensus.cc:695] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:36.364737 981 raft_consensus.cc:2270] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.364244 1359 consensus_queue.cc:237] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.365623 981 ts_tablet_manager.cc:1905] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.367309 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 74842fde5152419ca35f3950e465247a (table default.Table_9aeb558e2ae64fe2b45bef5d2b6a2625 [id=2bfacaaab9f347b1a77e5eb29a9c485e]). Expected version 1 got 0
I20250623 08:04:36.367867 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:36.368129 32056 catalog_manager.cc:5582] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:36.368566 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:36.369282 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]). Expected version 1 got 0
I20250623 08:04:36.366148 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet e22f9bae5f7248af99cec06ce6dbbd41 (table default.Table_dfc3da325f6849bfba88293a5eabb3e4 [id=ad93708e20ae4ef087d9a9b028d89806]) successfully deleted
I20250623 08:04:36.372494 342 tablet_bootstrap.cc:492] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:36.375876 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet d5cd316e6f4447be958611ab07ed31fc (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:36.376665 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 112) took 4441 ms. Trace:
I20250623 08:04:36.378562 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b19e5f2359a84c7ea02a0b25240a54cb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:36.377760 979 rpcz_store.cc:276] 0623 08:04:31.934945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.935889 (+ 944us) service_pool.cc:225] Handling call
0623 08:04:36.373541 (+4437652us) ts_tablet_manager.cc:1145] Deleting tablet 67c44697ecab4dd9b43f544a602897ca
0623 08:04:36.373731 (+ 190us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.374390 (+ 659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4437312}
I20250623 08:04:36.380721 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185c2bf7b9254947bb76b7fe0924559e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.381417 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b9e843ff1344a1aaacf038f15f64cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.381810 342 ts_tablet_manager.cc:1397] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
W20250623 08:04:36.382211 32453 leader_election.cc:343] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 70f9d21b8c144f9cada9bdbe0fbb1538
I20250623 08:04:36.383262 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_bc1251ce708a4467b5b1c71a1df307e0 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 1 got 0
I20250623 08:04:36.384159 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:36.384599 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:36.387802 981 ts_tablet_manager.cc:1918] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.388166 981 log.cc:1199] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/cde7df7d39a54114a4dcb28cfd1eb429
I20250623 08:04:36.388708 981 ts_tablet_manager.cc:1939] T cde7df7d39a54114a4dcb28cfd1eb429 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.389199 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:36.389479 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f83f68ebda7e4826bbb17278cd4ac5a8 (table default.Table_6acb69304c424830a635f22f12343b31 [id=c1782341d0d34662889cff00d557ff81]). Expected version 1 got 0
I20250623 08:04:36.389654 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:36.389950 32056 catalog_manager.cc:5582] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:36.390341 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 136e482995a24c5daec55a701f3091fa (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:36.390637 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 025365d2fa0a43dd983d395a05cc2af1 (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:36.387871 32053 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c4f705a7cbd940da941c460b61c5dece" } new_table_name: "default.Table_a774d79724e6498fa40937c65bf75b80" modify_external_catalogs: true
I20250623 08:04:36.390200 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 102) took 3958 ms. Trace:
I20250623 08:04:36.390897 981 rpcz_store.cc:276] 0623 08:04:32.432153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.432300 (+ 147us) service_pool.cc:225] Handling call
0623 08:04:36.361874 (+3929574us) ts_tablet_manager.cc:1145] Deleting tablet cde7df7d39a54114a4dcb28cfd1eb429
0623 08:04:36.361884 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.377665 (+ 15781us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.383388 (+ 5723us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.390150 (+ 6762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3929384}
I20250623 08:04:36.391690 657 raft_consensus.cc:491] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.391875 657 raft_consensus.cc:513] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.392207 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 103) took 3959 ms. Trace:
I20250623 08:04:36.392284 981 rpcz_store.cc:276] 0623 08:04:32.432266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.432525 (+ 259us) service_pool.cc:225] Handling call
0623 08:04:36.392078 (+3959553us) ts_tablet_manager.cc:1145] Deleting tablet c6a97b34e386443fa19a9601621b5017
0623 08:04:36.392089 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.392162 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3959334}
I20250623 08:04:36.392418 657 leader_election.cc:290] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.392554 981 tablet_replica.cc:331] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.391944 339 tablet_bootstrap.cc:492] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.392740 339 ts_tablet_manager.cc:1397] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.011s
I20250623 08:04:36.393204 32056 catalog_manager.cc:5582] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:36.393590 339 raft_consensus.cc:357] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.393784 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:36.394224 339 raft_consensus.cc:383] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.394374 339 raft_consensus.cc:738] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.394367 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_aaa3ca8d9bb0406981f0dfff8cbf9e3c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 4 got 0
I20250623 08:04:36.394666 339 consensus_queue.cc:260] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.395265 339 ts_tablet_manager.cc:1428] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:36.395567 339 tablet_bootstrap.cc:492] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.397284 339 tablet_bootstrap.cc:654] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.397698 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 771ecfc41e2b4007a4d213cec5b128c6 (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:36.397521 981 raft_consensus.cc:2241] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.397969 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:36.398198 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:36.398339 981 raft_consensus.cc:2270] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.398561 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:36.398914 981 ts_tablet_manager.cc:1905] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.405231 981 ts_tablet_manager.cc:1918] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.408615 981 log.cc:1199] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/ae7c2ac546114a819a543f1667fb7844
I20250623 08:04:36.409217 981 ts_tablet_manager.cc:1939] T ae7c2ac546114a819a543f1667fb7844 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.410461 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "70f9d21b8c144f9cada9bdbe0fbb1538" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.411926 339 tablet_bootstrap.cc:492] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.412127 339 ts_tablet_manager.cc:1397] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20250623 08:04:36.412797 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 109) took 4478 ms. Trace:
I20250623 08:04:36.412892 979 rpcz_store.cc:276] 0623 08:04:31.934559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.936263 (+ 1704us) service_pool.cc:225] Handling call
0623 08:04:36.412677 (+4476414us) ts_tablet_manager.cc:1145] Deleting tablet 2060a643d0da47efa868f52bdb331b1f
0623 08:04:36.412686 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.412777 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4476155}
I20250623 08:04:36.413026 339 raft_consensus.cc:357] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.413313 339 raft_consensus.cc:383] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.413442 339 raft_consensus.cc:738] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:36.402992 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 67c44697ecab4dd9b43f544a602897ca because the tablet was not found. No further retry: Not found: Tablet not found: 67c44697ecab4dd9b43f544a602897ca
I20250623 08:04:36.413700 339 consensus_queue.cc:260] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.414222 339 ts_tablet_manager.cc:1428] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250623 08:04:36.414201 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 2060a643d0da47efa868f52bdb331b1f because the tablet was not found. No further retry: Not found: Tablet not found: 2060a643d0da47efa868f52bdb331b1f
I20250623 08:04:36.414271 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:36.414554 339 tablet_bootstrap.cc:492] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.415016 32376 raft_consensus.cc:3058] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.415079 32056 catalog_manager.cc:5582] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:36.415534 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:36.415779 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_e279e539bb974bd28995fc3110dffc56 [id=670bededffc8499e8731b15f7e03380c]). Expected version 1 got 0
I20250623 08:04:36.416193 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:36.416577 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 48da1e4759bb4fea87a45ad8a428b6e7
W20250623 08:04:36.416652 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 077bb090b3ea4d029463304d7a6feee5 consensus state has no leader
I20250623 08:04:36.416795 848 consensus_queue.cc:1035] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250623 08:04:36.417035 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_738cccce8e6c47d683b9a7c569aa58e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.417263 32453 leader_election.cc:343] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 19536c16ff32460e9db571e4ce411bab
I20250623 08:04:36.417260 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_7c712a75a62f41dba90fcf504f4f752c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.417598 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_738cccce8e6c47d683b9a7c569aa58e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:36.417775 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c712a75a62f41dba90fcf504f4f752c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.418179 32345 tablet_service.cc:1468] Processing CreateTablet for tablet b720fdea539f423ba2d2aa9f89b09ae9 (DEFAULT_TABLE table=default.Table_59f2123c149e43569f3e3db991d1d09c [id=107d7d6ec5da4d29a8dbd45ba77eb017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.418530 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b720fdea539f423ba2d2aa9f89b09ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.418601 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e602d391526d4b0eb0dbf5189afbe075" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.418879 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet b84b8b8713774ab4896842feb9fb075e with delete_type TABLET_DATA_DELETED (replaced by tablet a3576624314d4e85b411aab9109683f7) from {username='slave'} at 127.0.0.1:35542
W20250623 08:04:36.419013 32053 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37b38cb54d6c45308358f556a692eabe consensus state has no leader
I20250623 08:04:36.419088 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 886fc0fa39c54bddb25ac01f68d95805) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.419569 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_e9f2ab1d95fb4341b5d05f6f5bd09f15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.420028 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9f2ab1d95fb4341b5d05f6f5bd09f15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.421072 32057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91] alter complete (version 1)
I20250623 08:04:36.422377 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.422542 32655 raft_consensus.cc:2466] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:36.423612 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet b61b02cf2904404f8c0ffef7d268fefc
I20250623 08:04:36.424410 32058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_712680c326284dd782d57dc0ee153879"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.425015 32058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_712680c326284dd782d57dc0ee153879 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:36.427233 32050 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23928d64bcef4c06af81698e688b7f94 consensus state has no leader
I20250623 08:04:36.427567 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.428033 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.431839 342 raft_consensus.cc:357] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.432126 342 raft_consensus.cc:383] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.432610 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 53) took 5385 ms. Trace:
I20250623 08:04:36.433187 342 raft_consensus.cc:738] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.433454 342 consensus_queue.cc:260] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.435999 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet b61b02cf2904404f8c0ffef7d268fefc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.436182 952 rpcz_store.cc:276] 0623 08:04:30.967222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.976097 (+ 8875us) service_pool.cc:225] Handling call
0623 08:04:36.140486 (+5164389us) ts_tablet_manager.cc:1145] Deleting tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa
0623 08:04:36.140496 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.308479 (+167983us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.350005 (+ 41526us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.352401 (+ 2396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5164089}
I20250623 08:04:36.440050 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 48da1e4759bb4fea87a45ad8a428b6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.440312 952 tablet_replica.cc:331] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:36.445631 339 tablet_bootstrap.cc:654] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:36.446218 32589 leader_election.cc:343] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.446337 32589 leader_election.cc:304] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:36.446854 1440 raft_consensus.cc:2747] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.447435 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_56c6dd0a0ee64ed088889c23db0fd2e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.448063 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56c6dd0a0ee64ed088889c23db0fd2e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.448343 1159 raft_consensus.cc:491] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.450789 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a0e4ca05337b448ca84bafd20b982a81" } new_table_name: "default.Table_0972bb6b77644e27b56f3e0007bc5fd4" modify_external_catalogs: true
I20250623 08:04:36.448100 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 104) took 3990 ms. Trace:
I20250623 08:04:36.451033 981 rpcz_store.cc:276] 0623 08:04:32.457442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.457488 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:36.392485 (+3934997us) ts_tablet_manager.cc:1145] Deleting tablet ae7c2ac546114a819a543f1667fb7844
0623 08:04:36.392493 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.402129 (+ 9636us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.405129 (+ 3000us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.448061 (+ 42932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3934703}
I20250623 08:04:36.451463 981 tablet_replica.cc:331] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.451709 981 raft_consensus.cc:2241] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.451996 981 pending_rounds.cc:70] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:36.452773 1303 raft_consensus.cc:491] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.452908 1303 raft_consensus.cc:513] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.453523 1303 leader_election.cc:290] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.454138 981 pending_rounds.cc:77] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7170727426991394816 op_type: NO_OP noop_request { }
I20250623 08:04:36.454480 981 raft_consensus.cc:2887] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:36.454900 981 raft_consensus.cc:2270] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.463905 981 ts_tablet_manager.cc:1905] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.464589 952 raft_consensus.cc:2241] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.465131 1159 raft_consensus.cc:513] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.466627 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_c67b6294376343b38de7a2aa4ed7bcea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.467212 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c67b6294376343b38de7a2aa4ed7bcea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.468497 32057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00] alter complete (version 3)
I20250623 08:04:36.470134 32703 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:36.473676 952 raft_consensus.cc:2270] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.474261 952 ts_tablet_manager.cc:1905] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.474984 342 ts_tablet_manager.cc:1428] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.061s user 0.002s sys 0.000s
I20250623 08:04:36.474978 32058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_56d841d4028b4bcf9282392e22379235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.475519 32058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56d841d4028b4bcf9282392e22379235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.479985 339 tablet_bootstrap.cc:492] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.480170 339 ts_tablet_manager.cc:1397] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.005s
I20250623 08:04:36.481084 339 raft_consensus.cc:357] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.481307 339 raft_consensus.cc:383] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.481441 339 raft_consensus.cc:738] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.481943 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet f5f58f55070344ff98a3808f70e0c755 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.482163 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 185c2bf7b9254947bb76b7fe0924559e (DEFAULT_TABLE table=default.Table_06b731510ae743608f1fbc4d250acf14 [id=7fbe15151173464ea0ee480720c58b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.482182 339 consensus_queue.cc:260] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.482491 981 ts_tablet_manager.cc:1918] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250623 08:04:36.483157 981 log.cc:1199] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/966a159e212040109e2842c15b69fb71
I20250623 08:04:36.483183 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185c2bf7b9254947bb76b7fe0924559e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.483656 981 ts_tablet_manager.cc:1939] T 966a159e212040109e2842c15b69fb71 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.484649 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 105) took 4026 ms. Trace:
I20250623 08:04:36.485163 342 tablet_bootstrap.cc:492] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:36.485589 981 rpcz_store.cc:276] 0623 08:04:32.457926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.457968 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:36.451401 (+3993433us) ts_tablet_manager.cc:1145] Deleting tablet 966a159e212040109e2842c15b69fb71
0623 08:04:36.451411 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.473906 (+ 22495us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.481812 (+ 7906us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.484616 (+ 2804us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3993215}
I20250623 08:04:36.486280 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 107) took 4023 ms. Trace:
I20250623 08:04:36.486722 981 rpcz_store.cc:276] 0623 08:04:32.462890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.462936 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:36.486144 (+4023208us) ts_tablet_manager.cc:1145] Deleting tablet 1a070895f4984493a3b9f6fa779797a9
0623 08:04:36.486153 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.486252 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4022997}
I20250623 08:04:36.487030 1159 leader_election.cc:290] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.487362 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 108) took 4024 ms. Trace:
I20250623 08:04:36.487802 981 rpcz_store.cc:276] 0623 08:04:32.463366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.463406 (+ 40us) service_pool.cc:225] Handling call
0623 08:04:36.487216 (+4023810us) ts_tablet_manager.cc:1145] Deleting tablet 3249605158634f4eab5b9d61357c383f
0623 08:04:36.487223 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.487329 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4023254}
I20250623 08:04:36.488229 981 tablet_replica.cc:331] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.488607 981 raft_consensus.cc:2241] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.489024 981 raft_consensus.cc:2270] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.489126 32492 tablet_service.cc:1468] Processing CreateTablet for tablet e7b9e843ff1344a1aaacf038f15f64cd (DEFAULT_TABLE table=default.Table_d74a4572f91e48d0b1c123e11ecf5b51 [id=e1e321dbfa254d21b11fc31f07d4de07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.489742 981 ts_tablet_manager.cc:1905] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.483035 339 ts_tablet_manager.cc:1428] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:36.490892 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]) successfully deleted
I20250623 08:04:36.491030 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef663ffdaf141aea5e6269b6b2e1c89" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.491200 32646 raft_consensus.cc:2466] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:36.492043 339 tablet_bootstrap.cc:492] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.493695 339 tablet_bootstrap.cc:654] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.495612 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 48da1e4759bb4fea87a45ad8a428b6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.497577 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 89a24019b797467e857b744d701fb1ed (DEFAULT_TABLE table=default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.498615 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_f933517ba72b4e44bd0432107e9a9f9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.499228 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f933517ba72b4e44bd0432107e9a9f9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.500932 32453 leader_election.cc:304] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:36.502861 339 tablet_bootstrap.cc:492] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.502864 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 74c1cdc83ffd4a09a163479ac58fd405 (DEFAULT_TABLE table=default.Table_fbb1fc5910454112b795f72c26f1af5b [id=6d6ee37c928745ef877266573fe021f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.503175 339 ts_tablet_manager.cc:1397] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20250623 08:04:36.503979 339 raft_consensus.cc:357] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.504160 339 raft_consensus.cc:383] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.504230 339 raft_consensus.cc:738] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.504307 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_59f2123c149e43569f3e3db991d1d09c" } modify_external_catalogs: true
I20250623 08:04:36.504464 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_59f2123c149e43569f3e3db991d1d09c" } modify_external_catalogs: true
I20250623 08:04:36.504457 339 consensus_queue.cc:260] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:36.506094 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90ceb0f8d5c94c4e9427042d8a35e16c consensus state has no leader
I20250623 08:04:36.506448 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e279e539bb974bd28995fc3110dffc56" } new_table_name: "default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec" modify_external_catalogs: true
W20250623 08:04:36.506898 32044 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 37b38cb54d6c45308358f556a692eabe consensus state has no leader
I20250623 08:04:36.509285 848 consensus_queue.cc:1035] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:36.509747 339 ts_tablet_manager.cc:1428] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:36.510093 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b9e843ff1344a1aaacf038f15f64cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.510102 339 tablet_bootstrap.cc:492] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.511873 339 tablet_bootstrap.cc:654] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.514957 1440 raft_consensus.cc:491] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.515103 1440 raft_consensus.cc:513] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.515681 1440 leader_election.cc:290] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:36.515782 981 ts_tablet_manager.cc:1918] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:36.516494 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc0dad440b494018bc0d6cfd40940bc1" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.518147 981 log.cc:1199] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/c7fcf583064f4538913d9157e93068dd
W20250623 08:04:36.518476 32589 leader_election.cc:343] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.518997 981 ts_tablet_manager.cc:1939] T c7fcf583064f4538913d9157e93068dd P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.520074 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 109) took 4055 ms. Trace:
I20250623 08:04:36.521744 981 rpcz_store.cc:276] 0623 08:04:32.464098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.464142 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:36.488178 (+4024036us) ts_tablet_manager.cc:1145] Deleting tablet c7fcf583064f4538913d9157e93068dd
0623 08:04:36.488186 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.505536 (+ 17350us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.515649 (+ 10113us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.520023 (+ 4374us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4023836}
I20250623 08:04:36.522330 32057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0] alter complete (version 1)
I20250623 08:04:36.522689 981 tablet_replica.cc:331] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.522826 657 raft_consensus.cc:2802] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:36.522943 657 raft_consensus.cc:491] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:36.523442 981 raft_consensus.cc:2241] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.523854 32057 master_service.cc:948] ReplaceTablet: received request to replace tablet 136e482995a24c5daec55a701f3091fa from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:36.524298 981 raft_consensus.cc:2270] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.526116 657 raft_consensus.cc:3058] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:36.526896 1413 tablet.cc:1722] T c63029399994480ba7ccabcadc1ee5e0 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:36.527236 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet b61b02cf2904404f8c0ffef7d268fefc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.527383 981 ts_tablet_manager.cc:1905] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.528263 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.528487 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.534145 32453 leader_election.cc:304] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:36.535382 32053 master_service.cc:948] ReplaceTablet: received request to replace tablet a34da529ffde4595a23fc6596b30e3e9 from {username='slave'} at 127.0.0.1:36116
W20250623 08:04:36.536777 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e898ede64e641519c8ee3ea1853a4b0 consensus state has no leader
I20250623 08:04:36.538211 1426 tablet.cc:1722] T c63029399994480ba7ccabcadc1ee5e0 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:36.539600 1159 raft_consensus.cc:2802] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:36.539716 1159 raft_consensus.cc:491] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:36.543179 1159 raft_consensus.cc:3058] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.544376 342 tablet_bootstrap.cc:654] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.544940 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet b720fdea539f423ba2d2aa9f89b09ae9
I20250623 08:04:36.546092 343 tablet_bootstrap.cc:492] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:36.546240 343 ts_tablet_manager.cc:1397] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.360s user 0.004s sys 0.000s
I20250623 08:04:36.546280 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c1cdc83ffd4a09a163479ac58fd405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.547703 32345 tablet_service.cc:1468] Processing CreateTablet for tablet f5f58f55070344ff98a3808f70e0c755 (DEFAULT_TABLE table=default.Table_57b13d90b3894f80b5a1cec576d2fb3a [id=431858068ed8423e91969be835dc8378]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.548103 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5f58f55070344ff98a3808f70e0c755. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.551434 32358 tablet_service.cc:1468] Processing CreateTablet for tablet a383d10d53b04d5699dd3c4bad8878c7 (DEFAULT_TABLE table=default.Table_68d8428833794f57b663cdae5b7d6cb3 [id=bcf8b85a65544390b35245e6ce6f7336]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.551836 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a383d10d53b04d5699dd3c4bad8878c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.552240 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 185c2bf7b9254947bb76b7fe0924559e (DEFAULT_TABLE table=default.Table_06b731510ae743608f1fbc4d250acf14 [id=7fbe15151173464ea0ee480720c58b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.552599 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185c2bf7b9254947bb76b7fe0924559e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.553154 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc0dad440b494018bc0d6cfd40940bc1" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.554018 32051 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c390976ce10f40d193a81eae44cd862f" } modify_external_catalogs: true
I20250623 08:04:36.554373 32051 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c390976ce10f40d193a81eae44cd862f" } modify_external_catalogs: true
I20250623 08:04:36.555013 32352 tablet_service.cc:1468] Processing CreateTablet for tablet c7f759b707ef46df99873b92c694ad6b (DEFAULT_TABLE table=default.Table_31ab57df769c4b2cb4e2cb74d69b4eb5 [id=92602c2df9c44292abbed7d6095922c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.555331 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f759b707ef46df99873b92c694ad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.558985 1441 raft_consensus.cc:491] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.559019 32361 tablet_service.cc:1468] Processing CreateTablet for tablet baccec88aefb4056be9578bca13f1591 (DEFAULT_TABLE table=default.Table_5afa01a4e2fa4b7c97e51a681da648d0 [id=0a1bbca429e6427581026c4600755351]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.559116 1441 raft_consensus.cc:513] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.559336 32361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baccec88aefb4056be9578bca13f1591. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.559798 1441 leader_election.cc:290] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:36.559914 1441 raft_consensus.cc:491] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.560019 1441 raft_consensus.cc:513] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.560190 343 raft_consensus.cc:357] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.560290 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet b720fdea539f423ba2d2aa9f89b09ae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.560338 343 raft_consensus.cc:383] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.560386 343 raft_consensus.cc:738] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.560513 1441 leader_election.cc:290] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:36.560555 343 consensus_queue.cc:260] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.561143 343 ts_tablet_manager.cc:1428] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:36.561515 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b80a91ae2c084020b5d3444b570b28f2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.562690 32347 tablet_service.cc:1468] Processing CreateTablet for tablet 0cca63991d5b4ef5bd8a9181d127ebe3 (DEFAULT_TABLE table=default.Table_da00c692e65345c496f4e52c106aa2e6 [id=5ea293756d514f06b8d749c615417969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.563014 32347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cca63991d5b4ef5bd8a9181d127ebe3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:36.564412 32589 leader_election.cc:343] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.564643 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a24019b797467e857b744d701fb1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.566286 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 8911308855cb4414be426dbcec592f4a (DEFAULT_TABLE table=default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.566681 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8911308855cb4414be426dbcec592f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.568213 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet b81263a9ee9d4da1ac721c3b2bd21248
I20250623 08:04:36.569206 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet b81263a9ee9d4da1ac721c3b2bd21248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.569552 1332 raft_consensus.cc:491] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.569705 1332 raft_consensus.cc:513] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.570037 32356 tablet_service.cc:1468] Processing CreateTablet for tablet 4e06ef06db2643779f93617e27f100dd (DEFAULT_TABLE table=default.Table_8a32f131cc9048a583663a64ed26a4af [id=e201b6e61b4641b9b4894576ac00fad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.570231 1332 leader_election.cc:290] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.570425 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e06ef06db2643779f93617e27f100dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.570871 32360 tablet_service.cc:1515] Processing DeleteTablet for tablet f5f58f55070344ff98a3808f70e0c755 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.573632 1359 consensus_queue.cc:1035] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20250623 08:04:36.574647 32360 tablet_service.cc:1468] Processing CreateTablet for tablet 89a24019b797467e857b744d701fb1ed (DEFAULT_TABLE table=default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.574695 32354 tablet_service.cc:1468] Processing CreateTablet for tablet e7b9e843ff1344a1aaacf038f15f64cd (DEFAULT_TABLE table=default.Table_d74a4572f91e48d0b1c123e11ecf5b51 [id=e1e321dbfa254d21b11fc31f07d4de07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.575023 32360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a24019b797467e857b744d701fb1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.575209 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b9e843ff1344a1aaacf038f15f64cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.579008 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 74c1cdc83ffd4a09a163479ac58fd405 (DEFAULT_TABLE table=default.Table_fbb1fc5910454112b795f72c26f1af5b [id=6d6ee37c928745ef877266573fe021f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.579420 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c1cdc83ffd4a09a163479ac58fd405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.583199 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 48da1e4759bb4fea87a45ad8a428b6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.583448 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet b61b02cf2904404f8c0ffef7d268fefc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.583457 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet b720fdea539f423ba2d2aa9f89b09ae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.583683 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet b81263a9ee9d4da1ac721c3b2bd21248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.584401 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 35ab9f77a81a4490abb0d2c0c69cd90f (table default.Table_92e23e1081544f8c9b99f14654981494 [id=bbf8e66e745141348c56d1cea5716898]) successfully deleted
I20250623 08:04:36.585335 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 113) took 4642 ms. Trace:
I20250623 08:04:36.585472 979 rpcz_store.cc:276] 0623 08:04:31.943008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.943246 (+ 238us) service_pool.cc:225] Handling call
0623 08:04:36.540665 (+4597419us) ts_tablet_manager.cc:1145] Deleting tablet 965f1f0a107b4653b6aac01e0d8e259a
0623 08:04:36.584911 (+ 44246us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.585314 (+ 403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4597196}
I20250623 08:04:36.586016 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 114) took 4642 ms. Trace:
I20250623 08:04:36.586119 979 rpcz_store.cc:276] 0623 08:04:31.943178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.943499 (+ 321us) service_pool.cc:225] Handling call
0623 08:04:36.585933 (+4642434us) ts_tablet_manager.cc:1145] Deleting tablet b980c746a3c042bfae90aa076acbb1df
0623 08:04:36.585943 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.586003 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4642244}
I20250623 08:04:36.586560 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 115) took 4637 ms. Trace:
I20250623 08:04:36.586683 979 rpcz_store.cc:276] 0623 08:04:31.948675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.948741 (+ 66us) service_pool.cc:225] Handling call
0623 08:04:36.586463 (+4637722us) ts_tablet_manager.cc:1145] Deleting tablet e9f8aa424bbb4aa8a907259270d46656
0623 08:04:36.586471 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.586546 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4637536}
I20250623 08:04:36.587246 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 116) took 4633 ms. Trace:
I20250623 08:04:36.587360 979 rpcz_store.cc:276] 0623 08:04:31.953773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.953836 (+ 63us) service_pool.cc:225] Handling call
0623 08:04:36.587091 (+4633255us) ts_tablet_manager.cc:1145] Deleting tablet ecb422550efd40cf82bc92fca5c25eda
0623 08:04:36.587103 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.587230 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4633045}
I20250623 08:04:36.590478 1359 raft_consensus.cc:491] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.591612 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 120) took 4540 ms. Trace:
I20250623 08:04:36.591545 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_06e661ea03304faaaa16113c730a6d91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:36.591739 979 rpcz_store.cc:276] 0623 08:04:32.051604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.059598 (+ 7994us) service_pool.cc:225] Handling call
0623 08:04:36.591511 (+4531913us) ts_tablet_manager.cc:1145] Deleting tablet 3249605158634f4eab5b9d61357c383f
0623 08:04:36.591519 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.591596 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4508804}
W20250623 08:04:36.592173 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06e661ea03304faaaa16113c730a6d91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.592216 979 tablet_replica.cc:331] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:36.592439 979 raft_consensus.cc:2241] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.592607 979 raft_consensus.cc:2270] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.593148 979 ts_tablet_manager.cc:1905] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.595304 1440 raft_consensus.cc:491] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.595443 1440 raft_consensus.cc:513] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.596015 1440 leader_election.cc:290] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:36.598565 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfd3e263464c4f2c8af9262acf211193" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:36.599341 32589 leader_election.cc:343] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.604072 1359 raft_consensus.cc:513] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.605072 952 ts_tablet_manager.cc:1918] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.606155 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21080ab069d74457a9f7fe3aabdb3c7c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:36.606978 32589 leader_election.cc:343] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.614430 339 tablet_bootstrap.cc:492] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.617179 339 ts_tablet_manager.cc:1397] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.107s user 0.006s sys 0.001s
W20250623 08:04:36.617251 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 965f1f0a107b4653b6aac01e0d8e259a because the tablet was not found. No further retry: Not found: Tablet not found: 965f1f0a107b4653b6aac01e0d8e259a
W20250623 08:04:36.617545 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b980c746a3c042bfae90aa076acbb1df because the tablet was not found. No further retry: Not found: Tablet not found: b980c746a3c042bfae90aa076acbb1df
W20250623 08:04:36.617761 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet e9f8aa424bbb4aa8a907259270d46656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9f8aa424bbb4aa8a907259270d46656 already in progress: creating tablet
W20250623 08:04:36.617987 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet ecb422550efd40cf82bc92fca5c25eda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecb422550efd40cf82bc92fca5c25eda already in progress: creating tablet
W20250623 08:04:36.618180 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 3249605158634f4eab5b9d61357c383f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3249605158634f4eab5b9d61357c383f already in progress: creating tablet
I20250623 08:04:36.618466 339 raft_consensus.cc:357] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.618722 339 raft_consensus.cc:383] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.618891 339 raft_consensus.cc:738] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.619196 339 consensus_queue.cc:260] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.619979 339 ts_tablet_manager.cc:1428] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:36.622506 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_32db9f68dd974fb8aff372a8eeccce3d" } new_table_name: "default.Table_78e064bbf114486f9cfa40945fb67bf2" modify_external_catalogs: true
I20250623 08:04:36.622978 32058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_7987e1a82b644bd69724bc61cd84445a" } modify_external_catalogs: true
I20250623 08:04:36.623176 32058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_7987e1a82b644bd69724bc61cd84445a" } modify_external_catalogs: true
I20250623 08:04:36.623939 1159 raft_consensus.cc:513] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.624568 1159 leader_election.cc:290] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.626356 339 tablet_bootstrap.cc:492] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.626673 952 log.cc:1199] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/7da84969a6464b5caeaf47af0cbaaa4d
I20250623 08:04:36.627511 952 ts_tablet_manager.cc:1939] T 7da84969a6464b5caeaf47af0cbaaa4d P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:36.627928 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 61) took 5652 ms. Trace:
I20250623 08:04:36.628072 952 rpcz_store.cc:276] 0623 08:04:30.975761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.976900 (+ 1139us) service_pool.cc:225] Handling call
0623 08:04:36.439088 (+5462188us) ts_tablet_manager.cc:1145] Deleting tablet 7da84969a6464b5caeaf47af0cbaaa4d
0623 08:04:36.439098 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.576726 (+137628us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.604985 (+ 28259us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.627911 (+ 22926us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3525,"tablet-delete.queue_time_us":5461998}
I20250623 08:04:36.628549 952 tablet_replica.cc:331] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:36.628772 952 raft_consensus.cc:2241] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.634647 1359 leader_election.cc:290] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.634891 343 tablet_bootstrap.cc:492] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:36.635792 657 raft_consensus.cc:513] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.636296 343 tablet_bootstrap.cc:654] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.636353 657 leader_election.cc:290] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.628080 339 tablet_bootstrap.cc:654] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.638991 981 ts_tablet_manager.cc:1918] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.639120 981 log.cc:1199] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/7da84969a6464b5caeaf47af0cbaaa4d
I20250623 08:04:36.639685 981 ts_tablet_manager.cc:1939] T 7da84969a6464b5caeaf47af0cbaaa4d P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.640057 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 110) took 4173 ms. Trace:
I20250623 08:04:36.640159 981 rpcz_store.cc:276] 0623 08:04:32.466634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.466680 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:36.522594 (+4055914us) ts_tablet_manager.cc:1145] Deleting tablet 7da84969a6464b5caeaf47af0cbaaa4d
0623 08:04:36.522606 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.620292 (+ 97686us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.638825 (+ 18533us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.640029 (+ 1204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4055669}
I20250623 08:04:36.640535 981 tablet_replica.cc:331] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.640739 981 raft_consensus.cc:2241] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.640909 981 raft_consensus.cc:2270] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.641471 981 ts_tablet_manager.cc:1905] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.644819 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712abfca1fa4489f882534c33b9db1be" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.645488 1332 raft_consensus.cc:491] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.645568 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffafd10f26740c9a5b6c9bdc3bb23a0" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.645639 1332 raft_consensus.cc:513] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.646201 1332 leader_election.cc:290] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.646798 32314 leader_election.cc:304] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:36.647068 1332 raft_consensus.cc:2802] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:36.647209 1332 raft_consensus.cc:695] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:36.647457 1332 consensus_queue.cc:237] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.650110 1506 raft_consensus.cc:491] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.650267 1506 raft_consensus.cc:513] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.651011 1506 leader_election.cc:290] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:36.651612 952 raft_consensus.cc:2270] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.652958 952 ts_tablet_manager.cc:1905] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.653664 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ea63263b9b466e922169eecafab747" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:36.654269 32589 leader_election.cc:343] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.656306 339 tablet_bootstrap.cc:492] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.656527 339 ts_tablet_manager.cc:1397] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250623 08:04:36.657790 339 raft_consensus.cc:357] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.658037 339 raft_consensus.cc:383] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.658146 339 raft_consensus.cc:738] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.658463 339 consensus_queue.cc:260] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.661486 339 ts_tablet_manager.cc:1428] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:36.661829 339 tablet_bootstrap.cc:492] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:36.662281 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 108846us
W20250623 08:04:36.663383 32589 leader_election.cc:343] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.663503 32589 leader_election.cc:304] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:36.663969 1506 raft_consensus.cc:2747] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.664227 339 tablet_bootstrap.cc:654] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.666290 32632 tablet_service.cc:1468] Processing CreateTablet for tablet 297bdf6c0e424190a35b55cbeefe7161 (DEFAULT_TABLE table=default.Table_a8ada1b74f854fe58c88fc3d90167bd3 [id=a3e24dd912cb43d19c4eea2b259b3d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.670795 981 ts_tablet_manager.cc:1918] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.672307 981 log.cc:1199] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/f0117cbb09dd4067836266c9d6a32adf
I20250623 08:04:36.672794 981 ts_tablet_manager.cc:1939] T f0117cbb09dd4067836266c9d6a32adf P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.673960 339 tablet_bootstrap.cc:492] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.674710 339 ts_tablet_manager.cc:1397] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250623 08:04:36.674978 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 111) took 4207 ms. Trace:
I20250623 08:04:36.675843 981 rpcz_store.cc:276] 0623 08:04:32.467035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.467073 (+ 38us) service_pool.cc:225] Handling call
0623 08:04:36.640473 (+4173400us) ts_tablet_manager.cc:1145] Deleting tablet f0117cbb09dd4067836266c9d6a32adf
0623 08:04:36.640483 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.666085 (+ 25602us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.670680 (+ 4595us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.674292 (+ 3612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4173188}
I20250623 08:04:36.675936 1359 raft_consensus.cc:491] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.676056 1359 raft_consensus.cc:513] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.675683 339 raft_consensus.cc:357] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.676297 339 raft_consensus.cc:383] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.676429 339 raft_consensus.cc:738] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.676532 1359 leader_election.cc:290] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.676721 339 consensus_queue.cc:260] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:36.677332 32453 consensus_peers.cc:487] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:36.677474 32453 leader_election.cc:343] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:36.677727 32453 consensus_peers.cc:487] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:36.678222 339 ts_tablet_manager.cc:1428] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:36.678623 339 tablet_bootstrap.cc:492] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.680320 339 tablet_bootstrap.cc:654] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.680526 981 tablet_replica.cc:331] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.681430 981 raft_consensus.cc:2241] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.681658 981 raft_consensus.cc:2270] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:36.684032 32314 leader_election.cc:343] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 965f1f0a107b4653b6aac01e0d8e259a
W20250623 08:04:36.677829 32453 leader_election.cc:343] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.684322 32453 leader_election.cc:304] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
W20250623 08:04:36.684365 32314 leader_election.cc:343] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: f3d32798168345edb9898751d083057c
I20250623 08:04:36.682206 981 ts_tablet_manager.cc:1905] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.695925 32377 raft_consensus.cc:3058] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.696202 981 ts_tablet_manager.cc:1918] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.696319 981 log.cc:1199] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/417a050eada4403ebbddff255fb6fb5a
I20250623 08:04:36.696816 981 ts_tablet_manager.cc:1939] T 417a050eada4403ebbddff255fb6fb5a P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.697649 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 112) took 4227 ms. Trace:
I20250623 08:04:36.697795 981 rpcz_store.cc:276] 0623 08:04:32.470420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.470472 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:36.680454 (+4209982us) ts_tablet_manager.cc:1145] Deleting tablet 417a050eada4403ebbddff255fb6fb5a
0623 08:04:36.680463 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.689510 (+ 9047us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.696076 (+ 6566us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.697606 (+ 1530us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4209642}
I20250623 08:04:36.698184 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 122) took 4189 ms. Trace:
I20250623 08:04:36.698282 981 rpcz_store.cc:276] 0623 08:04:32.508434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.508485 (+ 51us) service_pool.cc:225] Handling call
0623 08:04:36.698044 (+4189559us) ts_tablet_manager.cc:1145] Deleting tablet b80a91ae2c084020b5d3444b570b28f2
0623 08:04:36.698053 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.698146 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4167156}
I20250623 08:04:36.698583 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 134) took 3430 ms. Trace:
I20250623 08:04:36.698689 981 rpcz_store.cc:276] 0623 08:04:33.268252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.268304 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:36.698475 (+3430171us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d1c8a67e149478a0d69842a4eb5b2
0623 08:04:36.698483 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.698549 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3429962}
I20250623 08:04:36.699497 339 tablet_bootstrap.cc:492] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.699564 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.699710 339 ts_tablet_manager.cc:1397] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20250623 08:04:36.699756 32377 raft_consensus.cc:2466] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:36.700318 32377 raft_consensus.cc:3058] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.700491 339 raft_consensus.cc:357] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.700666 339 raft_consensus.cc:383] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.700747 339 raft_consensus.cc:738] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.700981 339 consensus_queue.cc:260] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.701819 339 ts_tablet_manager.cc:1428] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:36.702171 339 tablet_bootstrap.cc:492] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.667711 32628 tablet_service.cc:1468] Processing CreateTablet for tablet 886fc0fa39c54bddb25ac01f68d95805 (DEFAULT_TABLE table=default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.673440 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef663ffdaf141aea5e6269b6b2e1c89" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:36.702953 32655 raft_consensus.cc:3058] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:36.704021 32453 leader_election.cc:343] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 19536c16ff32460e9db571e4ce411bab
I20250623 08:04:36.704128 32453 leader_election.cc:304] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:36.704331 1359 raft_consensus.cc:2747] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.704476 1359 raft_consensus.cc:2747] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:36.705255 32631 tablet_service.cc:1468] Processing CreateTablet for tablet a3576624314d4e85b411aab9109683f7 (DEFAULT_TABLE table=default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.705492 32655 raft_consensus.cc:2466] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:36.674449 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a0bbafe1ea34ab5ae1b3de25b50d4a3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.705689 32620 tablet_service.cc:1468] Processing CreateTablet for tablet 4e3920ac85e54379abbcc42dd80db504 (DEFAULT_TABLE table=default.Table_4533668b8daa4ff3ad00ccc95e97e032 [id=08d03b1b488c4b52abebcef8b3d0f853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.705753 32646 raft_consensus.cc:2466] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:36.675287 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:36.706103 32650 raft_consensus.cc:3058] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:36.706084 1332 raft_consensus.cc:491] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.706208 1332 raft_consensus.cc:513] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.706882 1332 leader_election.cc:290] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.708736 32650 raft_consensus.cc:2466] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 2.
I20250623 08:04:36.685945 343 tablet_bootstrap.cc:492] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:36.710871 343 ts_tablet_manager.cc:1397] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250623 08:04:36.711524 343 raft_consensus.cc:357] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.711659 343 raft_consensus.cc:383] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.711702 343 raft_consensus.cc:738] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.711866 343 consensus_queue.cc:260] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.712308 343 ts_tablet_manager.cc:1428] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250623 08:04:36.714103 32052 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.1:45613 (request call id 155) took 1346 ms (client timeout 1000 ms). Trace:
I20250623 08:04:36.714658 32376 raft_consensus.cc:3058] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.714860 32381 raft_consensus.cc:3058] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.715639 32372 raft_consensus.cc:3058] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.718644 32378 raft_consensus.cc:3058] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.718938 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef663ffdaf141aea5e6269b6b2e1c89" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.719744 339 tablet_bootstrap.cc:654] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.721361 981 tablet_replica.cc:331] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.721663 981 raft_consensus.cc:2241] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.721845 981 raft_consensus.cc:2270] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.722039 1332 raft_consensus.cc:491] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.722205 1332 raft_consensus.cc:513] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.722609 981 ts_tablet_manager.cc:1905] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.722837 1332 leader_election.cc:290] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.723227 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21080ab069d74457a9f7fe3aabdb3c7c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.723757 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b80a91ae2c084020b5d3444b570b28f2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.724050 32518 raft_consensus.cc:2466] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:36.724586 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfd3e263464c4f2c8af9262acf211193" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.726641 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ea63263b9b466e922169eecafab747" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.727809 32376 raft_consensus.cc:3058] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.728174 32375 raft_consensus.cc:3058] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.728088 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a0bbafe1ea34ab5ae1b3de25b50d4a3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.729723 979 ts_tablet_manager.cc:1918] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:36.729847 979 log.cc:1199] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/1a070895f4984493a3b9f6fa779797a9
I20250623 08:04:36.730147 979 ts_tablet_manager.cc:1939] T 1a070895f4984493a3b9f6fa779797a9 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:36.730585 952 ts_tablet_manager.cc:1918] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.730921 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eef663ffdaf141aea5e6269b6b2e1c89" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:36.734126 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 0c337a37fcb141bca87212317d750de8 (DEFAULT_TABLE table=default.Table_739315d3797c465ca4ac459ddc3477e5 [id=3b957d976c904262895846aaabc9596e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.734304 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 28d79c96bd1c4ddfb4a1dd8a9d83c3fe (DEFAULT_TABLE table=default.Table_23e6e7311fd242248a3adec872f67c93 [id=d71308abdeed4e7b9e623282d10aa15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.735002 32630 tablet_service.cc:1468] Processing CreateTablet for tablet e13dc85076b34903b35310b14dc31085 (DEFAULT_TABLE table=default.Table_315a89d642a74362b8a04f8b82edd22f [id=7f7d25e8c0904f09b3cbd93034adcb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.735195 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 88f2a63b5a734c8f9b7a53361590245d (DEFAULT_TABLE table=default.Table_7eaaf57b62b345609a21707908207acd [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.735436 32636 tablet_service.cc:1468] Processing CreateTablet for tablet f080a682e532406a83712b6f3e6d2f7b (DEFAULT_TABLE table=default.Table_fea0e0f4beec4bddb07a0d24efe95aa4 [id=d869eced4268436fbf4ba0666f3b7790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.736302 32622 tablet_service.cc:1468] Processing CreateTablet for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (DEFAULT_TABLE table=default.Table_9c574fd66ea84e8eb1cae99bbb828288 [id=677e05a247254ed8b35326b17b9616e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.737752 32626 tablet_service.cc:1468] Processing CreateTablet for tablet bcf753ede17f429b82612ad12e13185c (DEFAULT_TABLE table=default.Table_9968742b8af441aa8db7bd5343fb4059 [id=0e357a45fc4547d59f1f886531c3ac2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.738672 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 1b213c8c685c43868df1e814c79fc61f (DEFAULT_TABLE table=default.Table_08ce069e682d42acac14ed36e986d074 [id=056178f3b8954cadb83dd80b9cd13aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.739967 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "283bebc7063d437ba3e541d7dee3c0b5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:36.740175 32379 raft_consensus.cc:3058] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:36.730716 952 log.cc:1199] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/966a159e212040109e2842c15b69fb71
I20250623 08:04:36.742230 339 tablet_bootstrap.cc:492] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.742417 339 ts_tablet_manager.cc:1397] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.003s
I20250623 08:04:36.742741 952 ts_tablet_manager.cc:1939] T 966a159e212040109e2842c15b69fb71 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:36.743218 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 62) took 5765 ms. Trace:
I20250623 08:04:36.743232 339 raft_consensus.cc:357] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.743464 339 raft_consensus.cc:383] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.743517 32379 raft_consensus.cc:2466] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 2.
I20250623 08:04:36.743570 339 raft_consensus.cc:738] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.743794 339 consensus_queue.cc:260] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.744838 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48da1e4759bb4fea87a45ad8a428b6e7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.746286 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 297bdf6c0e424190a35b55cbeefe7161 (DEFAULT_TABLE table=default.Table_a8ada1b74f854fe58c88fc3d90167bd3 [id=a3e24dd912cb43d19c4eea2b259b3d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.746663 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297bdf6c0e424190a35b55cbeefe7161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.748490 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:36.750540 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20e7a78cfc384033810603316b9e66bf" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.751260 32379 raft_consensus.cc:3058] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.743486 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet b720fdea539f423ba2d2aa9f89b09ae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.743429 32490 tablet_service.cc:1468] Processing CreateTablet for tablet 886fc0fa39c54bddb25ac01f68d95805 (DEFAULT_TABLE table=default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.752110 952 rpcz_store.cc:276] 0623 08:04:30.977316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.977951 (+ 635us) service_pool.cc:225] Handling call
0623 08:04:36.628504 (+5650553us) ts_tablet_manager.cc:1145] Deleting tablet 966a159e212040109e2842c15b69fb71
0623 08:04:36.628514 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.709422 (+ 80908us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.730498 (+ 21076us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.743198 (+ 12700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5650328}
I20250623 08:04:36.752303 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886fc0fa39c54bddb25ac01f68d95805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.752453 952 tablet_replica.cc:331] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:36.752628 952 raft_consensus.cc:2241] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.752864 952 pending_rounds.cc:70] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:36.752925 952 pending_rounds.cc:77] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7170727428795125760 op_type: NO_OP noop_request { }
I20250623 08:04:36.753028 952 raft_consensus.cc:2887] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:36.753111 952 raft_consensus.cc:2270] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.743605 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet b81263a9ee9d4da1ac721c3b2bd21248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:36.743918 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 297bdf6c0e424190a35b55cbeefe7161 (DEFAULT_TABLE table=default.Table_a8ada1b74f854fe58c88fc3d90167bd3 [id=a3e24dd912cb43d19c4eea2b259b3d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.754921 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdec6e9573b041e0a57d1e0e0239d97f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.744822 32486 tablet_service.cc:1468] Processing CreateTablet for tablet a3576624314d4e85b411aab9109683f7 (DEFAULT_TABLE table=default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.755136 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297bdf6c0e424190a35b55cbeefe7161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.755393 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3576624314d4e85b411aab9109683f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.745182 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 4e3920ac85e54379abbcc42dd80db504 (DEFAULT_TABLE table=default.Table_4533668b8daa4ff3ad00ccc95e97e032 [id=08d03b1b488c4b52abebcef8b3d0f853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.758458 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3920ac85e54379abbcc42dd80db504. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.758445 32372 raft_consensus.cc:3058] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:36.761313 32052 rpcz_store.cc:269] 0623 08:04:35.367556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.378974 (+ 11418us) service_pool.cc:225] Handling call
0623 08:04:35.392994 (+ 14020us) spinlock_profiling.cc:243] Waited 8.64 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f3b58 00007f19c7640176 00007f19c763fea5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:36.714056 (+1321062us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40131,"reported_tablets":173,"spinlock_wait_cycles":19869440}
I20250623 08:04:36.762148 32052 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a7cbd31e5c664fba885a13dcbd9105ab" } new_table_name: "default.Table_9426e757e60c4d3a8d540d2e94b241d1" modify_external_catalogs: true
I20250623 08:04:36.745852 32498 tablet_service.cc:1468] Processing CreateTablet for tablet 88f2a63b5a734c8f9b7a53361590245d (DEFAULT_TABLE table=default.Table_7eaaf57b62b345609a21707908207acd [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.763510 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f2a63b5a734c8f9b7a53361590245d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.738842 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]) successfully deleted
I20250623 08:04:36.766130 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 776b412b3f564a7581c6a5e1c84e8499 (table default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652]) successfully deleted
I20250623 08:04:36.766371 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 9918b15e48144ef0b8f3aedc2135be59 (table default.Table_c09ee28ff44c4c43835c2faa880a1c86 [id=371e56649fc6441c8aa52739de5aa077]) successfully deleted
I20250623 08:04:36.766571 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet cde7df7d39a54114a4dcb28cfd1eb429 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]) successfully deleted
W20250623 08:04:36.766829 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet c6a97b34e386443fa19a9601621b5017 because the tablet was not found. No further retry: Not found: Tablet not found: c6a97b34e386443fa19a9601621b5017
I20250623 08:04:36.746083 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 28d79c96bd1c4ddfb4a1dd8a9d83c3fe (DEFAULT_TABLE table=default.Table_23e6e7311fd242248a3adec872f67c93 [id=d71308abdeed4e7b9e623282d10aa15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.746492 32487 tablet_service.cc:1468] Processing CreateTablet for tablet 0c337a37fcb141bca87212317d750de8 (DEFAULT_TABLE table=default.Table_739315d3797c465ca4ac459ddc3477e5 [id=3b957d976c904262895846aaabc9596e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.747124 32495 tablet_service.cc:1468] Processing CreateTablet for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (DEFAULT_TABLE table=default.Table_9c574fd66ea84e8eb1cae99bbb828288 [id=677e05a247254ed8b35326b17b9616e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.767149 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet ae7c2ac546114a819a543f1667fb7844 (table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]) successfully deleted
I20250623 08:04:36.767216 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d79c96bd1c4ddfb4a1dd8a9d83c3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.767336 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 966a159e212040109e2842c15b69fb71 (table default.Table_84c686e72347490ebc29749f8c9352f9 [id=27884529aa7042ffb7eb7812892c6305]) successfully deleted
I20250623 08:04:36.767379 32487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c337a37fcb141bca87212317d750de8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:36.767526 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 1a070895f4984493a3b9f6fa779797a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a070895f4984493a3b9f6fa779797a9 already in progress: creating tablet
W20250623 08:04:36.767753 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 3249605158634f4eab5b9d61357c383f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3249605158634f4eab5b9d61357c383f already in progress: creating tablet
I20250623 08:04:36.767901 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet c7fcf583064f4538913d9157e93068dd (table default.Table_a65e08da5dbe4b408014e6c3b662450f [id=abda89a30971466ea9a1f410d424ef58]) successfully deleted
I20250623 08:04:36.768080 339 ts_tablet_manager.cc:1428] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250623 08:04:36.747776 32500 tablet_service.cc:1468] Processing CreateTablet for tablet e13dc85076b34903b35310b14dc31085 (DEFAULT_TABLE table=default.Table_315a89d642a74362b8a04f8b82edd22f [id=7f7d25e8c0904f09b3cbd93034adcb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.768687 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e13dc85076b34903b35310b14dc31085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.748095 32483 tablet_service.cc:1468] Processing CreateTablet for tablet f080a682e532406a83712b6f3e6d2f7b (DEFAULT_TABLE table=default.Table_fea0e0f4beec4bddb07a0d24efe95aa4 [id=d869eced4268436fbf4ba0666f3b7790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.770973 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc73f52f0bc43c5bbdca2b89deecb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.771137 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f080a682e532406a83712b6f3e6d2f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.773042 981 ts_tablet_manager.cc:1918] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:36.773201 981 log.cc:1199] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/f279b63b55c64e548ded27b02025c9f5
I20250623 08:04:36.773594 981 ts_tablet_manager.cc:1939] T f279b63b55c64e548ded27b02025c9f5 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.773957 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 138) took 3354 ms. Trace:
I20250623 08:04:36.774050 981 rpcz_store.cc:276] 0623 08:04:33.419331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.419547 (+ 216us) service_pool.cc:225] Handling call
0623 08:04:36.721270 (+3301723us) ts_tablet_manager.cc:1145] Deleting tablet f279b63b55c64e548ded27b02025c9f5
0623 08:04:36.721283 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.744302 (+ 23019us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.772902 (+ 28600us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.773896 (+ 994us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301451}
I20250623 08:04:36.774443 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 140) took 3321 ms. Trace:
I20250623 08:04:36.774513 981 rpcz_store.cc:276] 0623 08:04:33.452693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.453050 (+ 357us) service_pool.cc:225] Handling call
0623 08:04:36.774300 (+3321250us) ts_tablet_manager.cc:1145] Deleting tablet 9049e3d9b39f420bb51221808409fc54
0623 08:04:36.774307 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.774400 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3320464}
I20250623 08:04:36.748600 32488 tablet_service.cc:1468] Processing CreateTablet for tablet bcf753ede17f429b82612ad12e13185c (DEFAULT_TABLE table=default.Table_9968742b8af441aa8db7bd5343fb4059 [id=0e357a45fc4547d59f1f886531c3ac2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.775386 339 tablet_bootstrap.cc:492] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:36.749394 32453 leader_election.cc:304] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:36.751047 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 1b213c8c685c43868df1e814c79fc61f (DEFAULT_TABLE table=default.Table_08ce069e682d42acac14ed36e986d074 [id=056178f3b8954cadb83dd80b9cd13aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.779018 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcf753ede17f429b82612ad12e13185c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.779067 1507 raft_consensus.cc:2802] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:36.779157 1507 raft_consensus.cc:695] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:36.779371 1507 consensus_queue.cc:237] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.780720 339 tablet_bootstrap.cc:654] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.782115 32453 leader_election.cc:304] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:36.782534 981 tablet_replica.cc:331] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.782720 1508 raft_consensus.cc:2802] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:36.782831 1508 raft_consensus.cc:491] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:36.782899 1508 raft_consensus.cc:3058] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.783373 981 raft_consensus.cc:2241] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.783579 981 raft_consensus.cc:2270] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.785552 981 ts_tablet_manager.cc:1905] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.785501 1508 raft_consensus.cc:513] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.786029 1508 leader_election.cc:290] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.786605 32453 leader_election.cc:304] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:36.788791 1303 raft_consensus.cc:491] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.788996 339 tablet_bootstrap.cc:492] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:36.788973 1303 raft_consensus.cc:513] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.789153 339 ts_tablet_manager.cc:1397] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.004s
I20250623 08:04:36.789636 1303 leader_election.cc:290] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.789870 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 886fc0fa39c54bddb25ac01f68d95805 (DEFAULT_TABLE table=default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.790042 339 raft_consensus.cc:357] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.790246 339 raft_consensus.cc:383] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.790277 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886fc0fa39c54bddb25ac01f68d95805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.790343 339 raft_consensus.cc:738] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.790601 339 consensus_queue.cc:260] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.791168 339 ts_tablet_manager.cc:1428] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:36.791491 339 tablet_bootstrap.cc:492] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:36.795661 32589 leader_election.cc:343] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.795799 32589 leader_election.cc:304] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:36.796155 1506 raft_consensus.cc:2747] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.795656 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71fadb10e6134478b013358bba056389" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.796448 32653 raft_consensus.cc:2466] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:36.797006 1508 raft_consensus.cc:2802] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:36.797240 1508 raft_consensus.cc:695] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:36.797529 1508 consensus_queue.cc:237] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.798810 32358 tablet_service.cc:1468] Processing CreateTablet for tablet a3576624314d4e85b411aab9109683f7 (DEFAULT_TABLE table=default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.799235 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3576624314d4e85b411aab9109683f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.802680 339 tablet_bootstrap.cc:654] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.803192 32589 leader_election.cc:304] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:36.803610 1506 raft_consensus.cc:2802] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:36.803684 1506 raft_consensus.cc:491] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:36.803740 1506 raft_consensus.cc:3058] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.804831 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b213c8c685c43868df1e814c79fc61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.806331 1506 raft_consensus.cc:513] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.806612 981 ts_tablet_manager.cc:1918] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:36.806766 981 log.cc:1199] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/358cb6ac24414bdfbc32155e732f9ac8
I20250623 08:04:36.807188 981 ts_tablet_manager.cc:1939] T 358cb6ac24414bdfbc32155e732f9ac8 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:36.807230 1506 leader_election.cc:290] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:36.808535 32589 leader_election.cc:343] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.808641 32589 leader_election.cc:304] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:36.809024 1506 raft_consensus.cc:2747] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:36.810201 32589 leader_election.cc:343] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.810288 32589 leader_election.cc:304] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:36.810681 1506 raft_consensus.cc:2747] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.807310 32360 tablet_service.cc:1515] Processing DeleteTablet for tablet 228375592aee48a1aa21e1d16962b663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:24 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.807714 32361 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d8f38a6de1e45e7bed4abdced684c92 with delete_type TABLET_DATA_DELETED (replaced by tablet 31a67b70cbc64bb3bfd5a34865109a01) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.807998 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b80a91ae2c084020b5d3444b570b28f2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:36.808036 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 28d79c96bd1c4ddfb4a1dd8a9d83c3fe (DEFAULT_TABLE table=default.Table_23e6e7311fd242248a3adec872f67c93 [id=d71308abdeed4e7b9e623282d10aa15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.813016 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d79c96bd1c4ddfb4a1dd8a9d83c3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.814687 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a0bbafe1ea34ab5ae1b3de25b50d4a3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:36.814899 32653 raft_consensus.cc:3058] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.815826 1525 tablet.cc:1722] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:36.808640 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 4e3920ac85e54379abbcc42dd80db504 (DEFAULT_TABLE table=default.Table_4533668b8daa4ff3ad00ccc95e97e032 [id=08d03b1b488c4b52abebcef8b3d0f853]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.808873 32356 tablet_service.cc:1515] Processing DeleteTablet for tablet 726d80e05a344f27b5a289723cb5aed7 with delete_type TABLET_DATA_DELETED (replaced by tablet 432d1642c0e44119b5bc05858463a136) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.809219 32347 tablet_service.cc:1515] Processing DeleteTablet for tablet c0f87fd62f1545089d00757f8214593a with delete_type TABLET_DATA_DELETED (replaced by tablet 1f76ef24c69f4b17a373b242d71da8b9) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.809507 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ea81642050141a9be84f151a5785abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.809809 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 88f2a63b5a734c8f9b7a53361590245d (DEFAULT_TABLE table=default.Table_7eaaf57b62b345609a21707908207acd [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.810003 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet 67c44697ecab4dd9b43f544a602897ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:25 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.810170 32355 tablet_service.cc:1468] Processing CreateTablet for tablet e13dc85076b34903b35310b14dc31085 (DEFAULT_TABLE table=default.Table_315a89d642a74362b8a04f8b82edd22f [id=7f7d25e8c0904f09b3cbd93034adcb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.817234 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3920ac85e54379abbcc42dd80db504. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.817464 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e13dc85076b34903b35310b14dc31085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.817823 32653 raft_consensus.cc:2466] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:36.818454 32589 leader_election.cc:343] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.810447 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet ce1f3ef3690c4837b95b105733884f1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.810551 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 0c337a37fcb141bca87212317d750de8 (DEFAULT_TABLE table=default.Table_739315d3797c465ca4ac459ddc3477e5 [id=3b957d976c904262895846aaabc9596e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.823109 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 139) took 3370 ms. Trace:
I20250623 08:04:36.823282 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c337a37fcb141bca87212317d750de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.823227 981 rpcz_store.cc:276] 0623 08:04:33.452553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.453352 (+ 799us) service_pool.cc:225] Handling call
0623 08:04:36.782445 (+3329093us) ts_tablet_manager.cc:1145] Deleting tablet 358cb6ac24414bdfbc32155e732f9ac8
0623 08:04:36.782457 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.797558 (+ 15101us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.806468 (+ 8910us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.823069 (+ 16601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3325128}
I20250623 08:04:36.823565 981 tablet_replica.cc:331] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:36.823794 981 raft_consensus.cc:2241] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.824095 981 raft_consensus.cc:2270] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.824712 981 ts_tablet_manager.cc:1905] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.810168 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet a414144276f6484ea4690204752e2142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.826712 1366 raft_consensus.cc:491] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.810783 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 35ab9f77a81a4490abb0d2c0c69cd90f with delete_type TABLET_DATA_DELETED (replaced by tablet b1aa596950da49a898e3ed9bbad8c45e) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.811127 32349 tablet_service.cc:1515] Processing DeleteTablet for tablet 162308da898d4dde8ab5b1ab01d0f393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.811664 32348 tablet_service.cc:1468] Processing CreateTablet for tablet f080a682e532406a83712b6f3e6d2f7b (DEFAULT_TABLE table=default.Table_fea0e0f4beec4bddb07a0d24efe95aa4 [id=d869eced4268436fbf4ba0666f3b7790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.826891 1366 raft_consensus.cc:513] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.827277 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f080a682e532406a83712b6f3e6d2f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.827587 1366 leader_election.cc:290] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.828006 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3d32798168345edb9898751d083057c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.829555 342 tablet_bootstrap.cc:492] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:36.829700 342 ts_tablet_manager.cc:1397] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.354s user 0.003s sys 0.014s
I20250623 08:04:36.832520 848 consensus_queue.cc:1035] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:36.833441 848 consensus_queue.cc:1035] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:36.834019 32453 leader_election.cc:343] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.834502 32632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297bdf6c0e424190a35b55cbeefe7161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.834787 32620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e3920ac85e54379abbcc42dd80db504. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.834897 343 tablet_bootstrap.cc:492] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:36.836265 343 tablet_bootstrap.cc:654] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.837409 952 ts_tablet_manager.cc:1905] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.839213 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3576624314d4e85b411aab9109683f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.844301 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712abfca1fa4489f882534c33b9db1be" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.845091 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ffafd10f26740c9a5b6c9bdc3bb23a0" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.845983 32628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 886fc0fa39c54bddb25ac01f68d95805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.846001 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48da1e4759bb4fea87a45ad8a428b6e7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.846446 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20e7a78cfc384033810603316b9e66bf" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.855341 342 raft_consensus.cc:357] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.855531 342 raft_consensus.cc:383] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.855581 342 raft_consensus.cc:738] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.855821 342 consensus_queue.cc:260] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.856390 342 ts_tablet_manager.cc:1428] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250623 08:04:36.856698 342 tablet_bootstrap.cc:492] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:36.858026 342 tablet_bootstrap.cc:654] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.864425 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e06ef06db2643779f93617e27f100dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:36.864686 32453 consensus_peers.cc:487] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:36.864852 32453 leader_election.cc:343] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:36.865139 32453 leader_election.cc:343] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.865779 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdec6e9573b041e0a57d1e0e0239d97f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.866449 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b80a91ae2c084020b5d3444b570b28f2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:36.866600 32518 raft_consensus.cc:3058] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:36.869238 32518 raft_consensus.cc:2466] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:36.873955 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f2a63b5a734c8f9b7a53361590245d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.875015 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 7da84969a6464b5caeaf47af0cbaaa4d (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]) successfully deleted
I20250623 08:04:36.875279 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet f0117cbb09dd4067836266c9d6a32adf (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:36.875501 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 417a050eada4403ebbddff255fb6fb5a (table default.Table_24b73ca48b554c6698827e520b13b121 [id=2857dcb54e8b4109996cea363085a395]) successfully deleted
W20250623 08:04:36.875756 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet b80a91ae2c084020b5d3444b570b28f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b80a91ae2c084020b5d3444b570b28f2 already in progress: creating tablet
I20250623 08:04:36.879232 343 tablet_bootstrap.cc:492] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:36.879343 343 ts_tablet_manager.cc:1397] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.045s user 0.014s sys 0.005s
I20250623 08:04:36.879948 343 raft_consensus.cc:357] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.880090 343 raft_consensus.cc:383] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.880131 343 raft_consensus.cc:738] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.883096 343 consensus_queue.cc:260] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.883431 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 7da84969a6464b5caeaf47af0cbaaa4d (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]) successfully deleted
I20250623 08:04:36.883998 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 966a159e212040109e2842c15b69fb71 (table default.Table_84c686e72347490ebc29749f8c9352f9 [id=27884529aa7042ffb7eb7812892c6305]) successfully deleted
I20250623 08:04:36.884621 343 ts_tablet_manager.cc:1428] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:36.887239 32355 tablet_service.cc:1468] Processing CreateTablet for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (DEFAULT_TABLE table=default.Table_9c574fd66ea84e8eb1cae99bbb828288 [id=677e05a247254ed8b35326b17b9616e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.887720 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc73f52f0bc43c5bbdca2b89deecb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.888113 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.888732 32342 tablet_service.cc:1515] Processing DeleteTablet for tablet 205da880685b49b3b8669f73dc2c563b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.889674 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet e9f8aa424bbb4aa8a907259270d46656 with delete_type TABLET_DATA_DELETED (replaced by tablet a0e9b2f810014e81a29ca9b41cd3f8ba) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.890008 32350 tablet_service.cc:1468] Processing CreateTablet for tablet bcf753ede17f429b82612ad12e13185c (DEFAULT_TABLE table=default.Table_9968742b8af441aa8db7bd5343fb4059 [id=0e357a45fc4547d59f1f886531c3ac2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.890225 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb422550efd40cf82bc92fca5c25eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.891072 32342 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dc696236a15420e9666cca708068fb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.892359 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3af33890e8ff40c881af74580254633b" } modify_external_catalogs: true
I20250623 08:04:36.892547 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3af33890e8ff40c881af74580254633b" } modify_external_catalogs: true
I20250623 08:04:36.893843 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 20e7a78cfc384033810603316b9e66bf
I20250623 08:04:36.894495 32058 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e541bb700f4c4e2abc268babf435c6b0" } new_table_name: "default.Table_094f2e2ebe104774b6a713a5bbb7fd2f" modify_external_catalogs: true
W20250623 08:04:36.895859 32050 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42c861e29e1e4e549021dae7255a856c consensus state has no leader
I20250623 08:04:36.896241 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_c748f600d56f4374bdb31ad502ec7557"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:36.896368 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcf753ede17f429b82612ad12e13185c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:36.896775 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c748f600d56f4374bdb31ad502ec7557 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.900746 342 tablet_bootstrap.cc:492] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:36.900923 342 ts_tablet_manager.cc:1397] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.002s
I20250623 08:04:36.901612 342 raft_consensus.cc:357] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.901796 342 raft_consensus.cc:383] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.901760 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6ed4cb252b1452087f8ee53ec78d985" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.900308 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet e22f9bae5f7248af99cec06ce6dbbd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.902446 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 1b213c8c685c43868df1e814c79fc61f (DEFAULT_TABLE table=default.Table_08ce069e682d42acac14ed36e986d074 [id=056178f3b8954cadb83dd80b9cd13aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:36.902868 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b213c8c685c43868df1e814c79fc61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.904608 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 124) took 4829 ms. Trace:
I20250623 08:04:36.904714 979 rpcz_store.cc:276] 0623 08:04:32.074760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.074829 (+ 69us) service_pool.cc:225] Handling call
0623 08:04:36.592176 (+4517347us) ts_tablet_manager.cc:1145] Deleting tablet 1a070895f4984493a3b9f6fa779797a9
0623 08:04:36.592185 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.669881 (+ 77696us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.729610 (+ 59729us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.904586 (+174976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4509066}
I20250623 08:04:36.904933 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c337a37fcb141bca87212317d750de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.905058 979 tablet_replica.cc:331] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:36.905220 979 raft_consensus.cc:2241] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.905517 979 raft_consensus.cc:2270] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.905673 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet f279b63b55c64e548ded27b02025c9f5 (table default.Table_6e40e423312e4f9e95174f277b8897ce [id=7038d3807ece414f8a8e76bfe107c3b3]) successfully deleted
W20250623 08:04:36.906006 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 9049e3d9b39f420bb51221808409fc54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9049e3d9b39f420bb51221808409fc54 already in progress: creating tablet
I20250623 08:04:36.906033 979 ts_tablet_manager.cc:1905] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.906934 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 358cb6ac24414bdfbc32155e732f9ac8 (table default.Table_06b64498b6e148d5b86187f2ce7e4c31 [id=b827ea08f0b14c9aa2b2c3d02d15f150]) successfully deleted
I20250623 08:04:36.907091 32628 tablet_service.cc:1515] Processing DeleteTablet for tablet 20e7a78cfc384033810603316b9e66bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.907416 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 1a070895f4984493a3b9f6fa779797a9 (table default.Table_79f69fca0b25496e896d93fdd78e14b6 [id=cf095cc479d741b8b7c2cd352724713a]) successfully deleted
I20250623 08:04:36.908299 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28d79c96bd1c4ddfb4a1dd8a9d83c3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.910939 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e13dc85076b34903b35310b14dc31085. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.912667 979 ts_tablet_manager.cc:1918] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:36.912756 979 log.cc:1199] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/c7fcf583064f4538913d9157e93068dd
I20250623 08:04:36.913089 979 ts_tablet_manager.cc:1939] T c7fcf583064f4538913d9157e93068dd P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:36.913781 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 126) took 4821 ms. Trace:
I20250623 08:04:36.913853 979 rpcz_store.cc:276] 0623 08:04:32.092070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.092414 (+ 344us) service_pool.cc:225] Handling call
0623 08:04:36.905012 (+4812598us) ts_tablet_manager.cc:1145] Deleting tablet c7fcf583064f4538913d9157e93068dd
0623 08:04:36.905022 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.909608 (+ 4586us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.912572 (+ 2964us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.913763 (+ 1191us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4805865}
I20250623 08:04:36.914444 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet c7fcf583064f4538913d9157e93068dd (table default.Table_a65e08da5dbe4b408014e6c3b662450f [id=abda89a30971466ea9a1f410d424ef58]) successfully deleted
I20250623 08:04:36.914726 979 tablet_replica.cc:331] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:36.914923 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f2a63b5a734c8f9b7a53361590245d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.915170 32349 tablet_service.cc:1515] Processing DeleteTablet for tablet d0ee5a4fe03848229524cc0105226d8c with delete_type TABLET_DATA_DELETED (replaced by tablet 38ecd4c243b840f4b66d8e01078dca64) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:36.916486 1303 raft_consensus.cc:491] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.916611 1303 raft_consensus.cc:513] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.914932 979 raft_consensus.cc:2241] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.918084 979 raft_consensus.cc:2270] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.918552 979 ts_tablet_manager.cc:1905] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.919092 1303 leader_election.cc:290] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.920689 343 tablet_bootstrap.cc:492] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:36.901859 342 raft_consensus.cc:738] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.921962 343 tablet_bootstrap.cc:654] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.922514 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 136e482995a24c5daec55a701f3091fa
I20250623 08:04:36.922987 32057 catalog_manager.cc:6652] ReplaceTablet: tablet 136e482995a24c5daec55a701f3091fa deleted and replaced by tablet 3bfad68d1f20486997ea1b0a8d97206b
I20250623 08:04:36.921869 342 consensus_queue.cc:260] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.923657 32051 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 283bebc7063d437ba3e541d7dee3c0b5
I20250623 08:04:36.924520 32055 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 771ecfc41e2b4007a4d213cec5b128c6
I20250623 08:04:36.924904 32055 catalog_manager.cc:6652] ReplaceTablet: tablet 771ecfc41e2b4007a4d213cec5b128c6 deleted and replaced by tablet e41b15c89a744a698ca5eee2f143b4bc
I20250623 08:04:36.925385 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet a34da529ffde4595a23fc6596b30e3e9
I20250623 08:04:36.925805 32053 catalog_manager.cc:6652] ReplaceTablet: tablet a34da529ffde4595a23fc6596b30e3e9 deleted and replaced by tablet 1cee3de5953244f98428747bea912f77
I20250623 08:04:36.926066 342 ts_tablet_manager.cc:1428] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20250623 08:04:36.926788 952 ts_tablet_manager.cc:1918] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250623 08:04:36.927834 952 log.cc:1199] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f0117cbb09dd4067836266c9d6a32adf
I20250623 08:04:36.928886 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b213c8c685c43868df1e814c79fc61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.929337 342 tablet_bootstrap.cc:492] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:36.925614 1303 raft_consensus.cc:491] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.931406 1303 raft_consensus.cc:513] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.932245 952 ts_tablet_manager.cc:1939] T f0117cbb09dd4067836266c9d6a32adf P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:36.934427 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc73f52f0bc43c5bbdca2b89deecb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.935689 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 63) took 5957 ms. Trace:
I20250623 08:04:36.937748 343 tablet_bootstrap.cc:492] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:36.937868 343 ts_tablet_manager.cc:1397] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.002s
I20250623 08:04:36.937876 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcf753ede17f429b82612ad12e13185c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.936631 952 rpcz_store.cc:276] 0623 08:04:30.978385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.979024 (+ 639us) service_pool.cc:225] Handling call
0623 08:04:36.752413 (+5773389us) ts_tablet_manager.cc:1145] Deleting tablet f0117cbb09dd4067836266c9d6a32adf
0623 08:04:36.752421 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.871358 (+118937us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.925914 (+ 54556us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.935664 (+ 9750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5773218}
I20250623 08:04:36.938545 343 raft_consensus.cc:357] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.938694 343 raft_consensus.cc:383] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.938766 343 raft_consensus.cc:738] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.938997 343 consensus_queue.cc:260] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.939566 343 ts_tablet_manager.cc:1428] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:36.939831 343 tablet_bootstrap.cc:492] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:36.940272 952 tablet_replica.cc:331] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:36.942173 952 raft_consensus.cc:2241] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:36.942854 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f080a682e532406a83712b6f3e6d2f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:36.942914 1303 leader_election.cc:290] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.942292 32314 leader_election.cc:304] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:36.943630 343 tablet_bootstrap.cc:654] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.944044 1303 raft_consensus.cc:2747] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.943961 342 tablet_bootstrap.cc:654] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.944567 32314 leader_election.cc:304] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:36.944779 1303 raft_consensus.cc:2747] T 712abfca1fa4489f882534c33b9db1be P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.945003 32314 leader_election.cc:304] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:36.945597 1303 raft_consensus.cc:2747] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.945791 952 raft_consensus.cc:2270] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.947088 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 136e482995a24c5daec55a701f3091fa with delete_type TABLET_DATA_DELETED (replaced by tablet 3bfad68d1f20486997ea1b0a8d97206b) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.947227 952 ts_tablet_manager.cc:1905] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:36.949977 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71fadb10e6134478b013358bba056389" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:36.953924 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6ed4cb252b1452087f8ee53ec78d985" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.961723 32428 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:36.962044 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_71f2f0414e0840be832096955576cb2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.962659 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71f2f0414e0840be832096955576cb2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.964007 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_1ced320c61ec4355bb2686a5c84a55eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.964455 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ced320c61ec4355bb2686a5c84a55eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.965647 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_d87c4052e62d4c1c87fc25cff314319b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.966120 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d87c4052e62d4c1c87fc25cff314319b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.966219 979 ts_tablet_manager.cc:1918] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:36.967094 1447 raft_consensus.cc:491] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.967236 1447 raft_consensus.cc:513] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.967265 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_787415ae5937475f90f90000aa575d0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:36.967711 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_787415ae5937475f90f90000aa575d0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.967839 1447 leader_election.cc:290] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:36.968549 342 tablet_bootstrap.cc:492] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:36.968684 342 ts_tablet_manager.cc:1397] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
W20250623 08:04:36.969316 32052 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f217a8eb12014fcf89b9df4a2cddf5f6 consensus state has no leader
I20250623 08:04:36.969293 1525 tablet.cc:1722] T 74842fde5152419ca35f3950e465247a P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:36.969352 342 raft_consensus.cc:357] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.969491 342 raft_consensus.cc:383] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:36.969544 342 raft_consensus.cc:738] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:36.969715 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_3cbae8872bcd4af3a1a837940945fab6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:36.969722 342 consensus_queue.cc:260] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:36.970149 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cbae8872bcd4af3a1a837940945fab6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:36.970314 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a0bbafe1ea34ab5ae1b3de25b50d4a3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:36.971187 342 ts_tablet_manager.cc:1428] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:36.971211 32314 leader_election.cc:304] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:36.971508 1332 raft_consensus.cc:2747] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.971698 342 tablet_bootstrap.cc:492] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:36.971956 32314 leader_election.cc:304] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:36.972146 979 log.cc:1199] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b80a91ae2c084020b5d3444b570b28f2
I20250623 08:04:36.975493 979 ts_tablet_manager.cc:1939] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:36.978910 1526 log.cc:927] Time spent T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250623 08:04:36.980319 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 283bebc7063d437ba3e541d7dee3c0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.979682 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 131) took 4527 ms. Trace:
I20250623 08:04:36.981297 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 771ecfc41e2b4007a4d213cec5b128c6 with delete_type TABLET_DATA_DELETED (replaced by tablet e41b15c89a744a698ca5eee2f143b4bc) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.982156 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet a34da529ffde4595a23fc6596b30e3e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cee3de5953244f98428747bea912f77) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:36.980857 979 rpcz_store.cc:276] 0623 08:04:32.452518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.452589 (+ 71us) service_pool.cc:225] Handling call
0623 08:04:36.914687 (+4462098us) ts_tablet_manager.cc:1145] Deleting tablet b80a91ae2c084020b5d3444b570b28f2
0623 08:04:36.914696 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.945138 (+ 30442us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.965933 (+ 20795us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:36.979441 (+ 13508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4461840}
I20250623 08:04:36.985992 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 141) took 3930 ms. Trace:
I20250623 08:04:36.986984 979 rpcz_store.cc:276] 0623 08:04:33.055051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.055625 (+ 574us) service_pool.cc:225] Handling call
0623 08:04:36.983863 (+3928238us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d1c8a67e149478a0d69842a4eb5b2
0623 08:04:36.984159 (+ 296us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.985972 (+ 1813us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3928026}
W20250623 08:04:36.988487 32025 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 87036.7us
I20250623 08:04:36.990087 979 tablet_replica.cc:331] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:36.990378 342 tablet_bootstrap.cc:654] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:36.991832 979 raft_consensus.cc:2241] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:36.992974 1506 raft_consensus.cc:491] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.993110 1506 raft_consensus.cc:513] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:36.993162 979 raft_consensus.cc:2270] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:36.993681 1506 leader_election.cc:290] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:36.994289 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e9bf7cba2244703b7402cb4be1a005e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:36.994803 32589 leader_election.cc:343] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:36.995400 1332 raft_consensus.cc:2747] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.995913 32314 leader_election.cc:304] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:36.996250 1332 raft_consensus.cc:2747] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:36.997239 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdc9d47f2232403fa71e14a6fe6f33c3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.997850 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31ecc74317d449e5b557b07a4fd14895" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:36.998010 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31ecc74317d449e5b557b07a4fd14895" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.998596 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdc9d47f2232403fa71e14a6fe6f33c3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:36.999079 1332 raft_consensus.cc:491] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:36.999224 1332 raft_consensus.cc:513] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:36.999876 1332 leader_election.cc:290] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.000602 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 136e482995a24c5daec55a701f3091fa with delete_type TABLET_DATA_DELETED (replaced by tablet 3bfad68d1f20486997ea1b0a8d97206b) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.000819 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 283bebc7063d437ba3e541d7dee3c0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.000891 32314 leader_election.cc:304] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:37.000977 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 771ecfc41e2b4007a4d213cec5b128c6 with delete_type TABLET_DATA_DELETED (replaced by tablet e41b15c89a744a698ca5eee2f143b4bc) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.001075 1332 raft_consensus.cc:2747] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:37.001116 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet a34da529ffde4595a23fc6596b30e3e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cee3de5953244f98428747bea912f77) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.002446 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025365d2fa0a43dd983d395a05cc2af1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.002511 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "025365d2fa0a43dd983d395a05cc2af1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.002678 32650 raft_consensus.cc:2391] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:37.002153 979 ts_tablet_manager.cc:1905] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.003157 32314 leader_election.cc:304] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:37.003394 1332 raft_consensus.cc:2747] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:37.003772 32314 leader_election.cc:304] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:37.004031 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 20e7a78cfc384033810603316b9e66bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.004215 1332 raft_consensus.cc:2747] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:37.005112 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b80a91ae2c084020b5d3444b570b28f2 (table default.Table_d226f9dab24f45ae896579983b3d1a67 [id=e0cd5a7f63b443c2a3e86ec75a1e3778]) successfully deleted
I20250623 08:04:37.006037 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 8c01121f2e644d6aa6e8ec228bf75be9
W20250623 08:04:37.006479 32058 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59015204545e436cb558fe4465a67c97 consensus state has no leader
I20250623 08:04:37.006841 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6b0157583afb475b863719603a5ba38d" } new_table_name: "default.Table_1f98d716ca3541379adbced08214bfdd" modify_external_catalogs: true
I20250623 08:04:37.006997 32058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_3980cd736bd342f4bcd5cf289c1c8c25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.007555 32058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3980cd736bd342f4bcd5cf289c1c8c25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.010486 952 ts_tablet_manager.cc:1918] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.011677 952 log.cc:1199] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/175ee90133994154b7e4b58b0770db3b
I20250623 08:04:37.015327 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_b161fc2726dc4831988f5ac03028b089"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.015931 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b161fc2726dc4831988f5ac03028b089 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.017073 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 8c01121f2e644d6aa6e8ec228bf75be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:37.016855 952 ts_tablet_manager.cc:1939] T 175ee90133994154b7e4b58b0770db3b P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:37.019194 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 64) took 6039 ms. Trace:
I20250623 08:04:37.021286 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_c1548afc4cff490a94e8a03d5fea9339"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:37.021718 342 tablet_bootstrap.cc:492] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.021848 342 ts_tablet_manager.cc:1397] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
W20250623 08:04:37.021832 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1548afc4cff490a94e8a03d5fea9339 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.022601 342 raft_consensus.cc:357] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.022790 342 raft_consensus.cc:383] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.022847 342 raft_consensus.cc:738] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.023030 342 consensus_queue.cc:260] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.023625 342 ts_tablet_manager.cc:1428] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:37.023905 342 tablet_bootstrap.cc:492] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.025060 32055 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_83732a7d68bb412f9c26b799fc16e4ec" } new_table_name: "default.Table_79cb8b58ae4c493db6f796981a67564d" modify_external_catalogs: true
I20250623 08:04:37.025696 342 tablet_bootstrap.cc:654] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.028563 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_35a9f061c58a4dd1b8ad6e194d32bbde" } modify_external_catalogs: true
I20250623 08:04:37.028739 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_35a9f061c58a4dd1b8ad6e194d32bbde" } modify_external_catalogs: true
I20250623 08:04:37.029951 32053 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c300ef24f71d4bebaa367db14adb1b8a" } modify_external_catalogs: true
I20250623 08:04:37.030128 32053 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c300ef24f71d4bebaa367db14adb1b8a" } modify_external_catalogs: true
I20250623 08:04:37.030114 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 8c01121f2e644d6aa6e8ec228bf75be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.030586 32314 leader_election.cc:304] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:37.031421 32057 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b04e38d3da44301a450fe1fb5e2ada6 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:37.020025 952 rpcz_store.cc:276] 0623 08:04:30.979410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.980197 (+ 787us) service_pool.cc:225] Handling call
0623 08:04:36.939636 (+5959439us) ts_tablet_manager.cc:1145] Deleting tablet 175ee90133994154b7e4b58b0770db3b
0623 08:04:36.939947 (+ 311us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:36.975265 (+ 35318us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.009412 (+ 34147us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.019173 (+ 9761us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5959166}
I20250623 08:04:37.032289 32051 master_service.cc:948] ReplaceTablet: received request to replace tablet 771ecfc41e2b4007a4d213cec5b128c6 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:37.032410 952 tablet_replica.cc:331] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.032608 952 raft_consensus.cc:2241] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.034193 952 raft_consensus.cc:2270] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.038054 952 ts_tablet_manager.cc:1905] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:37.039706 32025 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 46566.5us
I20250623 08:04:37.040661 1332 raft_consensus.cc:3058] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.044024 1332 raft_consensus.cc:2747] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:37.059396 342 tablet_bootstrap.cc:492] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.059554 342 ts_tablet_manager.cc:1397] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.001s
I20250623 08:04:37.060317 342 raft_consensus.cc:357] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.060498 342 raft_consensus.cc:383] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.060565 342 raft_consensus.cc:738] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.060839 342 consensus_queue.cc:260] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.062161 342 ts_tablet_manager.cc:1428] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:37.062862 342 tablet_bootstrap.cc:492] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.071970 343 tablet_bootstrap.cc:492] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:37.072216 343 ts_tablet_manager.cc:1397] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.132s user 0.013s sys 0.040s
I20250623 08:04:37.072407 979 ts_tablet_manager.cc:1918] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.073642 343 raft_consensus.cc:357] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.073575 979 log.cc:1199] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/f279b63b55c64e548ded27b02025c9f5
I20250623 08:04:37.074214 343 raft_consensus.cc:383] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.074483 343 raft_consensus.cc:738] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.074863 343 consensus_queue.cc:260] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.075742 343 ts_tablet_manager.cc:1428] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:37.076469 343 tablet_bootstrap.cc:492] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:37.079396 343 tablet_bootstrap.cc:654] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.079522 979 ts_tablet_manager.cc:1939] T f279b63b55c64e548ded27b02025c9f5 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.080366 342 tablet_bootstrap.cc:654] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.082312 981 ts_tablet_manager.cc:1918] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.081959 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 146) took 3676 ms. Trace:
I20250623 08:04:37.083235 979 rpcz_store.cc:276] 0623 08:04:33.405094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.405163 (+ 69us) service_pool.cc:225] Handling call
0623 08:04:36.989746 (+3584583us) ts_tablet_manager.cc:1145] Deleting tablet f279b63b55c64e548ded27b02025c9f5
0623 08:04:36.989761 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.037219 (+ 47458us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.071565 (+ 34346us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.081712 (+ 10147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3583972}
I20250623 08:04:37.085273 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet f279b63b55c64e548ded27b02025c9f5 (table default.Table_6e40e423312e4f9e95174f277b8897ce [id=7038d3807ece414f8a8e76bfe107c3b3]) successfully deleted
I20250623 08:04:37.086205 979 tablet_replica.cc:331] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:37.087327 979 raft_consensus.cc:2241] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.088611 979 raft_consensus.cc:2270] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.089139 981 log.cc:1199] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/485538f7c63948818f20a25c02381519
I20250623 08:04:37.090201 981 ts_tablet_manager.cc:1939] T 485538f7c63948818f20a25c02381519 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:37.091710 979 ts_tablet_manager.cc:1905] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.096921 952 ts_tablet_manager.cc:1918] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.097074 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d98654c29b9c4e75af6a213cafd4c2c9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.097254 32650 raft_consensus.cc:2466] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:37.097746 952 log.cc:1199] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/5629d11c44e241d598f3e519b2fe1a44
I20250623 08:04:37.100939 952 ts_tablet_manager.cc:1939] T 5629d11c44e241d598f3e519b2fe1a44 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:37.101876 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_bcdbc5ed766f43ba91aaa4711f0b24f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:37.105832 32058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_52d445f76863453eb994aa3b0736a5f0" } modify_external_catalogs: true
I20250623 08:04:37.099814 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d98654c29b9c4e75af6a213cafd4c2c9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.105538 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 65) took 6124 ms. Trace:
I20250623 08:04:37.100191 32372 raft_consensus.cc:3058] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.100339 1525 tablet.cc:1722] T c63029399994480ba7ccabcadc1ee5e0 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:37.108728 342 tablet_bootstrap.cc:492] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.108863 342 ts_tablet_manager.cc:1397] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.000s
I20250623 08:04:37.109668 342 raft_consensus.cc:357] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.109830 342 raft_consensus.cc:383] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.109882 342 raft_consensus.cc:738] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.110069 342 consensus_queue.cc:260] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.110569 342 ts_tablet_manager.cc:1428] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:37.111003 32589 leader_election.cc:304] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:37.111199 32589 raft_consensus.cc:2684] T b80a91ae2c084020b5d3444b570b28f2 P 125745849172449ba4be8244d61fdcb3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:37.111200 342 tablet_bootstrap.cc:492] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.106135 952 rpcz_store.cc:276] 0623 08:04:30.980723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.981520 (+ 797us) service_pool.cc:225] Handling call
0623 08:04:37.032359 (+6050839us) ts_tablet_manager.cc:1145] Deleting tablet 5629d11c44e241d598f3e519b2fe1a44
0623 08:04:37.032368 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.071973 (+ 39605us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.096305 (+ 24332us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.105397 (+ 9092us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6050654}
I20250623 08:04:37.112426 32453 leader_election.cc:304] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:37.112915 1359 raft_consensus.cc:2802] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:37.113103 1359 raft_consensus.cc:491] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.113672 952 tablet_replica.cc:331] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.113431 1359 raft_consensus.cc:3058] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.115967 952 raft_consensus.cc:2241] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.117892 952 raft_consensus.cc:2270] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:37.122339 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcdbc5ed766f43ba91aaa4711f0b24f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.123052 342 tablet_bootstrap.cc:654] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.125340 952 ts_tablet_manager.cc:1905] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.130688 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 141) took 3629 ms. Trace:
I20250623 08:04:37.131467 981 rpcz_store.cc:276] 0623 08:04:33.501059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.505511 (+ 4452us) service_pool.cc:225] Handling call
0623 08:04:36.823499 (+3317988us) ts_tablet_manager.cc:1145] Deleting tablet 485538f7c63948818f20a25c02381519
0623 08:04:36.823507 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.065310 (+241803us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.082197 (+ 16887us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.130658 (+ 48461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3317033}
I20250623 08:04:37.133590 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 485538f7c63948818f20a25c02381519 (table default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]) successfully deleted
I20250623 08:04:37.136516 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 142) took 3605 ms. Trace:
I20250623 08:04:37.136623 1359 raft_consensus.cc:513] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.137241 981 rpcz_store.cc:276] 0623 08:04:33.530642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.530697 (+ 55us) service_pool.cc:225] Handling call
0623 08:04:37.134606 (+3603909us) ts_tablet_manager.cc:1145] Deleting tablet dc0dad440b494018bc0d6cfd40940bc1
0623 08:04:37.135170 (+ 564us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.136274 (+ 1104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3602871}
W20250623 08:04:37.139686 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet dc0dad440b494018bc0d6cfd40940bc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc0dad440b494018bc0d6cfd40940bc1 already in progress: creating tablet
I20250623 08:04:37.142160 1359 leader_election.cc:290] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.148808 342 tablet_bootstrap.cc:492] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.148883 981 tablet_replica.cc:331] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:37.149062 342 ts_tablet_manager.cc:1397] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.001s
I20250623 08:04:37.149789 342 raft_consensus.cc:357] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.150053 342 raft_consensus.cc:383] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.150194 342 raft_consensus.cc:738] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.150602 342 consensus_queue.cc:260] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.152273 342 ts_tablet_manager.cc:1428] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:37.147902 979 ts_tablet_manager.cc:1918] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.154924 981 raft_consensus.cc:2241] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:37.155778 981 raft_consensus.cc:2270] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.156602 981 ts_tablet_manager.cc:1905] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.158030 342 tablet_bootstrap.cc:492] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.157593 979 log.cc:1199] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/358cb6ac24414bdfbc32155e732f9ac8
I20250623 08:04:37.160869 343 tablet_bootstrap.cc:492] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:37.161325 979 ts_tablet_manager.cc:1939] T 358cb6ac24414bdfbc32155e732f9ac8 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.161841 343 ts_tablet_manager.cc:1397] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.085s user 0.013s sys 0.009s
I20250623 08:04:37.162941 981 ts_tablet_manager.cc:1918] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.167846 981 log.cc:1199] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/effd4fff43fe4d4fb36458986e1dfce1
I20250623 08:04:37.167659 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 147) took 3759 ms. Trace:
I20250623 08:04:37.168370 981 ts_tablet_manager.cc:1939] T effd4fff43fe4d4fb36458986e1dfce1 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:37.168243 343 raft_consensus.cc:357] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.168428 979 rpcz_store.cc:276] 0623 08:04:33.408049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.408108 (+ 59us) service_pool.cc:225] Handling call
0623 08:04:37.085701 (+3677593us) ts_tablet_manager.cc:1145] Deleting tablet 358cb6ac24414bdfbc32155e732f9ac8
0623 08:04:37.085716 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.128700 (+ 42984us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.147390 (+ 18690us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.167628 (+ 20238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3677149}
I20250623 08:04:37.170089 343 raft_consensus.cc:383] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.170835 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 358cb6ac24414bdfbc32155e732f9ac8 (table default.Table_06b64498b6e148d5b86187f2ce7e4c31 [id=b827ea08f0b14c9aa2b2c3d02d15f150]) successfully deleted
I20250623 08:04:37.170957 343 raft_consensus.cc:738] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.172828 343 consensus_queue.cc:260] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.174670 952 ts_tablet_manager.cc:1918] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.175314 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 148) took 3723 ms. Trace:
I20250623 08:04:37.178059 1506 raft_consensus.cc:491] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.178563 1506 raft_consensus.cc:513] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.178856 343 ts_tablet_manager.cc:1428] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20250623 08:04:37.179188 1506 leader_election.cc:290] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:37.179891 979 rpcz_store.cc:276] 0623 08:04:33.451544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.451603 (+ 59us) service_pool.cc:225] Handling call
0623 08:04:37.171397 (+3719794us) ts_tablet_manager.cc:1145] Deleting tablet 9049e3d9b39f420bb51221808409fc54
0623 08:04:37.171409 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.175294 (+ 3885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3719293}
W20250623 08:04:37.180405 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9049e3d9b39f420bb51221808409fc54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9049e3d9b39f420bb51221808409fc54 already in progress: creating tablet
I20250623 08:04:37.180979 979 tablet_replica.cc:331] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:37.181154 979 raft_consensus.cc:2241] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.181308 979 raft_consensus.cc:2270] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.181790 343 tablet_bootstrap.cc:492] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:37.175069 342 tablet_bootstrap.cc:654] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.183306 952 log.cc:1199] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/417a050eada4403ebbddff255fb6fb5a
I20250623 08:04:37.186257 952 ts_tablet_manager.cc:1939] T 417a050eada4403ebbddff255fb6fb5a P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:37.186582 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:37.187234 32589 leader_election.cc:343] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.181856 979 ts_tablet_manager.cc:1905] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.192945 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 143) took 3606 ms. Trace:
I20250623 08:04:37.193127 981 rpcz_store.cc:276] 0623 08:04:33.586959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.587024 (+ 65us) service_pool.cc:225] Handling call
0623 08:04:37.140829 (+3553805us) ts_tablet_manager.cc:1145] Deleting tablet effd4fff43fe4d4fb36458986e1dfce1
0623 08:04:37.141066 (+ 237us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.159898 (+ 18832us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.162859 (+ 2961us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.192925 (+ 30066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3553536}
I20250623 08:04:37.193671 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet effd4fff43fe4d4fb36458986e1dfce1 (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]) successfully deleted
I20250623 08:04:37.194075 981 tablet_replica.cc:331] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:37.194322 981 raft_consensus.cc:2241] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:37.195418 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 66) took 6213 ms. Trace:
I20250623 08:04:37.195565 952 rpcz_store.cc:276] 0623 08:04:30.981975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.982614 (+ 639us) service_pool.cc:225] Handling call
0623 08:04:37.112191 (+6129577us) ts_tablet_manager.cc:1145] Deleting tablet 417a050eada4403ebbddff255fb6fb5a
0623 08:04:37.112203 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.146467 (+ 34264us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.174563 (+ 28096us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.195402 (+ 20839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6129063}
I20250623 08:04:37.195966 952 tablet_replica.cc:331] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.196025 1525 tablet.cc:1722] T 0db9480bb07343fe935fe9bfc98bca77 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:37.196250 952 raft_consensus.cc:2241] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.196463 952 raft_consensus.cc:2270] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.197360 32453 leader_election.cc:304] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:37.197746 1507 raft_consensus.cc:2802] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:37.197901 1507 raft_consensus.cc:695] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:37.197990 952 ts_tablet_manager.cc:1905] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.198156 1507 consensus_queue.cc:237] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.194705 981 raft_consensus.cc:2270] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.202498 343 tablet_bootstrap.cc:654] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.203434 981 ts_tablet_manager.cc:1905] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.206643 339 tablet_bootstrap.cc:492] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.206845 339 ts_tablet_manager.cc:1397] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.415s user 0.026s sys 0.096s
I20250623 08:04:37.207638 339 raft_consensus.cc:357] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.207877 339 raft_consensus.cc:383] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.207999 339 raft_consensus.cc:738] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.208353 339 consensus_queue.cc:260] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.209213 339 ts_tablet_manager.cc:1428] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:37.211261 342 tablet_bootstrap.cc:492] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.211453 342 ts_tablet_manager.cc:1397] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.002s
I20250623 08:04:37.212002 32058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_52d445f76863453eb994aa3b0736a5f0" } modify_external_catalogs: true
I20250623 08:04:37.212627 979 ts_tablet_manager.cc:1918] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.212759 979 log.cc:1199] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/485538f7c63948818f20a25c02381519
I20250623 08:04:37.213114 979 ts_tablet_manager.cc:1939] T 485538f7c63948818f20a25c02381519 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.213434 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 149) took 3713 ms. Trace:
I20250623 08:04:37.213513 979 rpcz_store.cc:276] 0623 08:04:33.499556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.501688 (+ 2132us) service_pool.cc:225] Handling call
0623 08:04:37.180912 (+3679224us) ts_tablet_manager.cc:1145] Deleting tablet 485538f7c63948818f20a25c02381519
0623 08:04:37.180922 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.191796 (+ 10874us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.208582 (+ 16786us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.213416 (+ 4834us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3636629}
I20250623 08:04:37.214004 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 485538f7c63948818f20a25c02381519 (table default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]) successfully deleted
I20250623 08:04:37.215322 339 tablet_bootstrap.cc:492] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.216161 979 tablet_replica.cc:331] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:37.216379 979 raft_consensus.cc:2241] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.216537 979 raft_consensus.cc:2270] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.217065 979 ts_tablet_manager.cc:1905] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.223583 339 tablet_bootstrap.cc:654] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.224603 32052 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_aaa3ca8d9bb0406981f0dfff8cbf9e3c" } new_table_name: "default.Table_4e8bf0e3037248cfb48d25ab98e5468c" modify_external_catalogs: true
I20250623 08:04:37.226907 32453 leader_election.cc:304] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:37.227319 32453 consensus_peers.cc:487] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:37.227438 1359 consensus_queue.cc:1035] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:37.227586 32453 consensus_peers.cc:487] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a070895f4984493a3b9f6fa779797a9. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:37.228096 342 raft_consensus.cc:357] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.228932 342 raft_consensus.cc:383] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.228976 342 raft_consensus.cc:738] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.229147 342 consensus_queue.cc:260] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.229281 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet f0117cbb09dd4067836266c9d6a32adf (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:37.229542 32378 raft_consensus.cc:3058] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.230412 342 ts_tablet_manager.cc:1428] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250623 08:04:37.230741 342 tablet_bootstrap.cc:492] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.233011 339 tablet_bootstrap.cc:492] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.235002 952 ts_tablet_manager.cc:1918] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.235193 952 log.cc:1199] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/19536c16ff32460e9db571e4ce411bab
I20250623 08:04:37.235769 1359 raft_consensus.cc:491] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.235882 339 ts_tablet_manager.cc:1397] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.003s
I20250623 08:04:37.235890 1359 raft_consensus.cc:513] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.236791 339 raft_consensus.cc:357] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.238131 343 tablet_bootstrap.cc:492] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:37.238302 343 ts_tablet_manager.cc:1397] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.002s
I20250623 08:04:37.238817 1359 leader_election.cc:290] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.239244 343 raft_consensus.cc:357] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.239506 343 raft_consensus.cc:383] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.239588 343 raft_consensus.cc:738] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.239858 343 consensus_queue.cc:260] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.227460 1508 raft_consensus.cc:2802] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:37.240340 1508 raft_consensus.cc:491] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.240403 1508 raft_consensus.cc:3058] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.240475 343 ts_tablet_manager.cc:1428] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:37.240746 343 tablet_bootstrap.cc:492] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:37.243774 1508 raft_consensus.cc:513] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.245231 1508 leader_election.cc:290] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.246022 339 raft_consensus.cc:383] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.246153 339 raft_consensus.cc:738] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.246879 343 tablet_bootstrap.cc:654] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.247161 339 consensus_queue.cc:260] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.248631 339 ts_tablet_manager.cc:1428] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250623 08:04:37.248941 342 tablet_bootstrap.cc:654] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.251820 32059 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:37.257560 339 tablet_bootstrap.cc:492] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.258651 1506 raft_consensus.cc:491] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.258802 1506 raft_consensus.cc:513] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.259387 1506 leader_election.cc:290] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:37.259992 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736eb56dae194df2af90bf33d3294a4e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:37.260449 32589 leader_election.cc:343] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.261996 952 ts_tablet_manager.cc:1939] T 19536c16ff32460e9db571e4ce411bab P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:37.263244 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 67) took 6280 ms. Trace:
I20250623 08:04:37.264753 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75a6ebe0d6b0463b8ddcb1a091a2cc6f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.264961 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 175ee90133994154b7e4b58b0770db3b (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]) successfully deleted
I20250623 08:04:37.265693 339 tablet_bootstrap.cc:654] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.266234 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 5629d11c44e241d598f3e519b2fe1a44 (table default.Table_4d4535cab44545839eb3a559385e395a [id=bb849e86d7084959830c68207fb99061]) successfully deleted
I20250623 08:04:37.266911 32372 raft_consensus.cc:3058] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.267831 343 tablet_bootstrap.cc:492] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:37.267940 343 ts_tablet_manager.cc:1397] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20250623 08:04:37.268591 32381 raft_consensus.cc:3058] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.268651 343 raft_consensus.cc:357] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.269167 343 raft_consensus.cc:383] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.269222 343 raft_consensus.cc:738] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.269383 979 ts_tablet_manager.cc:1918] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:37.269449 343 consensus_queue.cc:260] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.269677 979 log.cc:1199] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/dc0dad440b494018bc0d6cfd40940bc1
I20250623 08:04:37.270031 343 ts_tablet_manager.cc:1428] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:37.270315 343 tablet_bootstrap.cc:492] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:37.270814 979 ts_tablet_manager.cc:1939] T dc0dad440b494018bc0d6cfd40940bc1 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.272003 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 150) took 3758 ms. Trace:
I20250623 08:04:37.272442 343 tablet_bootstrap.cc:654] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.272276 979 rpcz_store.cc:276] 0623 08:04:33.513663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.513733 (+ 70us) service_pool.cc:225] Handling call
0623 08:04:37.216106 (+3702373us) ts_tablet_manager.cc:1145] Deleting tablet dc0dad440b494018bc0d6cfd40940bc1
0623 08:04:37.216118 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.240151 (+ 24033us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.268998 (+ 28847us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.271985 (+ 2987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3671502}
I20250623 08:04:37.273367 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet dc0dad440b494018bc0d6cfd40940bc1 (table default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]) successfully deleted
I20250623 08:04:37.273545 979 tablet_replica.cc:331] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:37.274087 979 raft_consensus.cc:2241] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.274711 979 raft_consensus.cc:2270] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.274881 952 rpcz_store.cc:276] 0623 08:04:30.983164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.983666 (+ 502us) service_pool.cc:225] Handling call
0623 08:04:37.195921 (+6212255us) ts_tablet_manager.cc:1145] Deleting tablet 19536c16ff32460e9db571e4ce411bab
0623 08:04:37.195930 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.212122 (+ 16192us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.234900 (+ 22778us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.263224 (+ 28324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6212079}
I20250623 08:04:37.275768 952 tablet_replica.cc:331] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.276837 952 raft_consensus.cc:2241] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:37.276816 979 ts_tablet_manager.cc:1905] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.276640 342 tablet_bootstrap.cc:492] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.277155 342 ts_tablet_manager.cc:1397] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.005s
I20250623 08:04:37.279238 343 tablet_bootstrap.cc:492] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:37.279443 343 ts_tablet_manager.cc:1397] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20250623 08:04:37.279870 342 raft_consensus.cc:357] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.280241 343 raft_consensus.cc:357] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.280323 952 raft_consensus.cc:2270] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.280508 343 raft_consensus.cc:383] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.280596 343 raft_consensus.cc:738] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.281266 952 ts_tablet_manager.cc:1905] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.285013 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 6b04e38d3da44301a450fe1fb5e2ada6
I20250623 08:04:37.280411 342 raft_consensus.cc:383] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.285259 342 raft_consensus.cc:738] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.285526 342 consensus_queue.cc:260] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.285658 32057 catalog_manager.cc:6652] ReplaceTablet: tablet 6b04e38d3da44301a450fe1fb5e2ada6 deleted and replaced by tablet 7f416cdb596a4ed5a302acd07bb84b9c
I20250623 08:04:37.286019 32057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6acb69304c424830a635f22f12343b31" } modify_external_catalogs: true
I20250623 08:04:37.286231 32057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6acb69304c424830a635f22f12343b31" } modify_external_catalogs: true
I20250623 08:04:37.286134 342 ts_tablet_manager.cc:1428] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250623 08:04:37.287201 342 tablet_bootstrap.cc:492] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.287320 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b04e38d3da44301a450fe1fb5e2ada6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f416cdb596a4ed5a302acd07bb84b9c) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:37.287724 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b04e38d3da44301a450fe1fb5e2ada6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f416cdb596a4ed5a302acd07bb84b9c) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.287906 339 tablet_bootstrap.cc:492] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.288098 339 ts_tablet_manager.cc:1397] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.003s
I20250623 08:04:37.289016 339 raft_consensus.cc:357] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.289222 339 raft_consensus.cc:383] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.289324 339 raft_consensus.cc:738] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.288949 343 consensus_queue.cc:260] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.289671 339 consensus_queue.cc:260] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.290309 339 ts_tablet_manager.cc:1428] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:37.290303 343 ts_tablet_manager.cc:1428] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250623 08:04:37.290660 339 tablet_bootstrap.cc:492] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.290781 343 tablet_bootstrap.cc:492] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:37.292335 339 tablet_bootstrap.cc:654] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.292691 343 tablet_bootstrap.cc:654] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.296507 1555 tablet.cc:1722] T 0db9480bb07343fe935fe9bfc98bca77 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:37.304728 339 tablet_bootstrap.cc:492] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.304955 339 ts_tablet_manager.cc:1397] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250623 08:04:37.305742 339 raft_consensus.cc:357] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.306097 342 tablet_bootstrap.cc:654] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.306383 339 raft_consensus.cc:383] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.307157 339 raft_consensus.cc:738] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.307439 339 consensus_queue.cc:260] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.307448 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3188d5c072e04adea5dae4c5ec7a71c8" } new_table_name: "default.Table_50fb8135a67a4f3cbf9250135b3ded35" modify_external_catalogs: true
I20250623 08:04:37.310249 339 ts_tablet_manager.cc:1428] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:37.311010 339 tablet_bootstrap.cc:492] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.313594 952 ts_tablet_manager.cc:1918] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.316589 339 tablet_bootstrap.cc:654] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.318835 952 log.cc:1199] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/1e47b10f46b8402e94354ebca573c194
I20250623 08:04:37.319420 952 ts_tablet_manager.cc:1939] T 1e47b10f46b8402e94354ebca573c194 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:37.319691 342 tablet_bootstrap.cc:492] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.320351 342 ts_tablet_manager.cc:1397] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.003s
I20250623 08:04:37.321137 342 raft_consensus.cc:357] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.321362 342 raft_consensus.cc:383] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.321465 342 raft_consensus.cc:738] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.321707 342 consensus_queue.cc:260] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.322224 342 ts_tablet_manager.cc:1428] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:37.322551 342 tablet_bootstrap.cc:492] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.324142 342 tablet_bootstrap.cc:654] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.331239 342 tablet_bootstrap.cc:492] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.332175 342 ts_tablet_manager.cc:1397] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20250623 08:04:37.331676 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 68) took 6347 ms. Trace:
I20250623 08:04:37.333767 952 rpcz_store.cc:276] 0623 08:04:30.984000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:30.984491 (+ 491us) service_pool.cc:225] Handling call
0623 08:04:37.275718 (+6291227us) ts_tablet_manager.cc:1145] Deleting tablet 1e47b10f46b8402e94354ebca573c194
0623 08:04:37.275729 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.285097 (+ 9368us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.313506 (+ 28409us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.331659 (+ 18153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6291023}
I20250623 08:04:37.334678 952 tablet_replica.cc:331] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.335747 339 tablet_bootstrap.cc:492] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.336073 952 raft_consensus.cc:2241] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:37.336293 339 ts_tablet_manager.cc:1397] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250623 08:04:37.336686 952 raft_consensus.cc:2270] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.337939 952 ts_tablet_manager.cc:1905] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.338181 339 raft_consensus.cc:357] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.333567 342 raft_consensus.cc:357] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.342901 342 raft_consensus.cc:383] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.343035 342 raft_consensus.cc:738] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.343320 342 consensus_queue.cc:260] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.343999 342 ts_tablet_manager.cc:1428] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250623 08:04:37.344127 339 raft_consensus.cc:383] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.344485 342 tablet_bootstrap.cc:492] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.344609 339 raft_consensus.cc:738] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.344975 339 consensus_queue.cc:260] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.346875 342 tablet_bootstrap.cc:654] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.347644 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 207c75acfed64082a885b47ae5353c9d (DEFAULT_TABLE table=default.Table_c67b6294376343b38de7a2aa4ed7bcea [id=898fc58a25554c09867e70728f4a7508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.347726 339 ts_tablet_manager.cc:1428] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250623 08:04:37.348652 339 tablet_bootstrap.cc:492] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.348841 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 207c75acfed64082a885b47ae5353c9d (DEFAULT_TABLE table=default.Table_c67b6294376343b38de7a2aa4ed7bcea [id=898fc58a25554c09867e70728f4a7508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.349192 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 207c75acfed64082a885b47ae5353c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.352991 339 tablet_bootstrap.cc:654] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.358848 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71fadb10e6134478b013358bba056389" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:37.359072 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71fadb10e6134478b013358bba056389" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:37.359232 32646 raft_consensus.cc:3058] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.360294 979 ts_tablet_manager.cc:1918] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.360569 979 log.cc:1199] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/effd4fff43fe4d4fb36458986e1dfce1
I20250623 08:04:37.361351 952 ts_tablet_manager.cc:1918] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:37.361471 952 log.cc:1199] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/5fdc9e53da264a5ca1f70e1846c906f0
I20250623 08:04:37.361800 32646 raft_consensus.cc:2466] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:37.361863 952 ts_tablet_manager.cc:1939] T 5fdc9e53da264a5ca1f70e1846c906f0 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:37.362783 32372 raft_consensus.cc:3058] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.363317 979 ts_tablet_manager.cc:1939] T effd4fff43fe4d4fb36458986e1dfce1 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.363982 32378 raft_consensus.cc:3058] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.364674 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 151) took 3777 ms. Trace:
I20250623 08:04:37.365293 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 417a050eada4403ebbddff255fb6fb5a (table default.Table_24b73ca48b554c6698827e520b13b121 [id=2857dcb54e8b4109996cea363085a395]) successfully deleted
I20250623 08:04:37.364929 979 rpcz_store.cc:276] 0623 08:04:33.587705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.587773 (+ 68us) service_pool.cc:225] Handling call
0623 08:04:37.273307 (+3685534us) ts_tablet_manager.cc:1145] Deleting tablet effd4fff43fe4d4fb36458986e1dfce1
0623 08:04:37.273317 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.347790 (+ 74473us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.359917 (+ 12127us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.364654 (+ 4737us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3632505}
I20250623 08:04:37.365839 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet effd4fff43fe4d4fb36458986e1dfce1 (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]) successfully deleted
I20250623 08:04:37.366164 979 tablet_replica.cc:331] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:37.367744 979 raft_consensus.cc:2241] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.368443 979 raft_consensus.cc:2270] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.371331 979 ts_tablet_manager.cc:1905] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.375955 339 tablet_bootstrap.cc:492] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.376667 339 ts_tablet_manager.cc:1397] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250623 08:04:37.375257 1553 tablet.cc:1722] T 0db9480bb07343fe935fe9bfc98bca77 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:37.378949 339 raft_consensus.cc:357] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.379339 339 raft_consensus.cc:383] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.379532 339 raft_consensus.cc:738] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.379817 339 consensus_queue.cc:260] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.380443 339 ts_tablet_manager.cc:1428] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:37.380792 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 78) took 5937 ms. Trace:
I20250623 08:04:37.381135 339 tablet_bootstrap.cc:492] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.381264 952 rpcz_store.cc:276] 0623 08:04:31.443145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.443197 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:37.334601 (+5891404us) ts_tablet_manager.cc:1145] Deleting tablet 5fdc9e53da264a5ca1f70e1846c906f0
0623 08:04:37.334610 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.341464 (+ 6854us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.361237 (+ 19773us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.380773 (+ 19536us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5887528}
I20250623 08:04:37.381630 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (DEFAULT_TABLE table=default.Table_56d841d4028b4bcf9282392e22379235 [id=123fe4cae52542ddbde750a8de19320f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.382424 952 tablet_replica.cc:331] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.382937 952 raft_consensus.cc:2241] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:37.387065 952 raft_consensus.cc:2270] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.387288 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (DEFAULT_TABLE table=default.Table_56d841d4028b4bcf9282392e22379235 [id=123fe4cae52542ddbde750a8de19320f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.385166 342 tablet_bootstrap.cc:492] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.387694 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eebc32fdfa44abcbd6f04a636d02c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.387725 342 ts_tablet_manager.cc:1397] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.002s
I20250623 08:04:37.388572 342 raft_consensus.cc:357] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.389194 32350 tablet_service.cc:1468] Processing CreateTablet for tablet f0e24d8d890d4d1db051f172b3ec96ec (DEFAULT_TABLE table=default.Table_56c6dd0a0ee64ed088889c23db0fd2e9 [id=4199d682d171458ab79e4a36807c7f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.389480 32627 tablet_service.cc:1468] Processing CreateTablet for tablet f0e24d8d890d4d1db051f172b3ec96ec (DEFAULT_TABLE table=default.Table_56c6dd0a0ee64ed088889c23db0fd2e9 [id=4199d682d171458ab79e4a36807c7f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.389765 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e24d8d890d4d1db051f172b3ec96ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.390877 339 tablet_bootstrap.cc:654] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.391006 342 raft_consensus.cc:383] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.391494 342 raft_consensus.cc:738] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.391767 342 consensus_queue.cc:260] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.392352 342 ts_tablet_manager.cc:1428] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:37.393554 342 tablet_bootstrap.cc:492] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.396296 342 tablet_bootstrap.cc:654] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.396495 979 ts_tablet_manager.cc:1918] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:37.391402 952 ts_tablet_manager.cc:1905] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.397001 979 log.cc:1199] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/a25bdc9b7e48467fba3771365e58bcbc
I20250623 08:04:37.397998 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 3fcd0382328f4b0f9ac510c83eb1f68f (DEFAULT_TABLE table=default.Table_e9f2ab1d95fb4341b5d05f6f5bd09f15 [id=f0a8652a629545a9a9bc4c22aba0da74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.398190 979 ts_tablet_manager.cc:1939] T a25bdc9b7e48467fba3771365e58bcbc P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.398326 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fcd0382328f4b0f9ac510c83eb1f68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.399231 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 169) took 3044 ms. Trace:
I20250623 08:04:37.399509 979 rpcz_store.cc:276] 0623 08:04:34.354825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.354872 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:37.365852 (+3010980us) ts_tablet_manager.cc:1145] Deleting tablet a25bdc9b7e48467fba3771365e58bcbc
0623 08:04:37.365969 (+ 117us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.385816 (+ 19847us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.396047 (+ 10231us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.399128 (+ 3081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2998934}
I20250623 08:04:37.399452 343 tablet_bootstrap.cc:492] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:37.401638 979 tablet_replica.cc:331] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:37.401973 343 ts_tablet_manager.cc:1397] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.111s user 0.003s sys 0.000s
I20250623 08:04:37.402840 979 raft_consensus.cc:2241] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.403532 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 36d21fe528f5410eacd655bf15306b98 (DEFAULT_TABLE table=default.Table_a50378b31cda420a8791443844eed5f2 [id=b83148d602ab4c19a5fb1b4072b626e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.403743 979 raft_consensus.cc:2270] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.404332 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 492ffaeaaa314dcfb7512d8f23946a67 (DEFAULT_TABLE table=default.Table_712680c326284dd782d57dc0ee153879 [id=407f4156ebee45ecb729ad3b3d5f26e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.404837 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]) successfully deleted
I20250623 08:04:37.405531 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 36d21fe528f5410eacd655bf15306b98 (DEFAULT_TABLE table=default.Table_a50378b31cda420a8791443844eed5f2 [id=b83148d602ab4c19a5fb1b4072b626e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.405541 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 3fcd0382328f4b0f9ac510c83eb1f68f (DEFAULT_TABLE table=default.Table_e9f2ab1d95fb4341b5d05f6f5bd09f15 [id=f0a8652a629545a9a9bc4c22aba0da74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.406493 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 492ffaeaaa314dcfb7512d8f23946a67 (DEFAULT_TABLE table=default.Table_712680c326284dd782d57dc0ee153879 [id=407f4156ebee45ecb729ad3b3d5f26e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.406859 979 ts_tablet_manager.cc:1905] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.406646 343 raft_consensus.cc:357] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.407737 343 raft_consensus.cc:383] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.408401 342 tablet_bootstrap.cc:492] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.408211 343 raft_consensus.cc:738] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.409022 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36d21fe528f5410eacd655bf15306b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.410036 342 ts_tablet_manager.cc:1397] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20250623 08:04:37.409847 343 consensus_queue.cc:260] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.410852 342 raft_consensus.cc:357] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.411046 342 raft_consensus.cc:383] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.411123 342 raft_consensus.cc:738] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.411352 342 consensus_queue.cc:260] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.412261 339 tablet_bootstrap.cc:492] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.412376 339 ts_tablet_manager.cc:1397] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250623 08:04:37.413053 339 raft_consensus.cc:357] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.413201 339 raft_consensus.cc:383] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.413245 339 raft_consensus.cc:738] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.413453 339 consensus_queue.cc:260] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.416186 339 ts_tablet_manager.cc:1428] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:37.416455 339 tablet_bootstrap.cc:492] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.421658 343 ts_tablet_manager.cc:1428] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20250623 08:04:37.422571 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 492ffaeaaa314dcfb7512d8f23946a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.409839 952 ts_tablet_manager.cc:1918] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.423206 952 log.cc:1199] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f3d32798168345edb9898751d083057c
I20250623 08:04:37.423694 952 ts_tablet_manager.cc:1939] T f3d32798168345edb9898751d083057c P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:37.423789 339 tablet_bootstrap.cc:654] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.424959 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 77) took 5982 ms. Trace:
I20250623 08:04:37.425612 952 rpcz_store.cc:276] 0623 08:04:31.442858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.444794 (+ 1936us) service_pool.cc:225] Handling call
0623 08:04:37.382341 (+5937547us) ts_tablet_manager.cc:1145] Deleting tablet f3d32798168345edb9898751d083057c
0623 08:04:37.382350 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.400896 (+ 18546us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.409708 (+ 8812us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.424927 (+ 15219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5924501}
I20250623 08:04:37.426048 952 tablet_replica.cc:331] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.426321 952 raft_consensus.cc:2241] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.426564 952 raft_consensus.cc:2270] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.427232 952 ts_tablet_manager.cc:1905] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.429373 1447 raft_consensus.cc:491] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.429513 1447 raft_consensus.cc:513] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.430090 1447 leader_election.cc:290] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.425508 342 ts_tablet_manager.cc:1428] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250623 08:04:37.430610 342 tablet_bootstrap.cc:492] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.438154 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e9bf7cba2244703b7402cb4be1a005e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.438628 339 tablet_bootstrap.cc:492] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.438843 339 ts_tablet_manager.cc:1397] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20250623 08:04:37.439113 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736eb56dae194df2af90bf33d3294a4e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.439317 32514 raft_consensus.cc:2466] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:37.439639 339 raft_consensus.cc:357] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.439898 339 raft_consensus.cc:383] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.439859 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.439996 339 raft_consensus.cc:738] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:37.440002 32453 consensus_peers.cc:487] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:37.440295 32453 leader_election.cc:343] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.440282 339 consensus_queue.cc:260] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.440979 339 ts_tablet_manager.cc:1428] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:37.441275 339 tablet_bootstrap.cc:492] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:37.441303 32453 consensus_peers.cc:487] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:37.441540 32453 leader_election.cc:343] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.443197 343 tablet_bootstrap.cc:492] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:37.444070 32349 tablet_service.cc:1468] Processing CreateTablet for tablet 30dfa986dfd44658a0ffb07796030520 (DEFAULT_TABLE table=default.Table_738cccce8e6c47d683b9a7c569aa58e3 [id=20baa502b37c4ec1904f52ddeaf99c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.444618 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 30dfa986dfd44658a0ffb07796030520 (DEFAULT_TABLE table=default.Table_738cccce8e6c47d683b9a7c569aa58e3 [id=20baa502b37c4ec1904f52ddeaf99c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.444977 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30dfa986dfd44658a0ffb07796030520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.445688 339 tablet_bootstrap.cc:654] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:37.446554 32453 leader_election.cc:343] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:37.447521 32453 consensus_peers.cc:487] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f279b63b55c64e548ded27b02025c9f5. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:37.448606 952 ts_tablet_manager.cc:1918] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.448814 952 log.cc:1199] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/38ecd4c243b840f4b66d8e01078dca64
W20250623 08:04:37.449121 32453 leader_election.cc:343] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.449286 952 ts_tablet_manager.cc:1939] T 38ecd4c243b840f4b66d8e01078dca64 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:37.449368 32453 consensus_peers.cc:487] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358cb6ac24414bdfbc32155e732f9ac8. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:37.452739 1359 consensus_queue.cc:1035] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.452772 979 ts_tablet_manager.cc:1918] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.453681 339 tablet_bootstrap.cc:492] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.453357 979 log.cc:1199] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/8ffafd10f26740c9a5b6c9bdc3bb23a0
I20250623 08:04:37.453871 339 ts_tablet_manager.cc:1397] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
W20250623 08:04:37.453979 32453 leader_election.cc:343] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.454763 339 raft_consensus.cc:357] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.454972 339 raft_consensus.cc:383] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.455072 339 raft_consensus.cc:738] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.455345 339 consensus_queue.cc:260] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.455513 979 ts_tablet_manager.cc:1939] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.456066 32358 tablet_service.cc:1468] Processing CreateTablet for tablet dcfe68d4e6ae470d816ac4c494dcaa51 (DEFAULT_TABLE table=default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe [id=f77e622d3eb44f4fa3a284b08eefc109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.456442 339 ts_tablet_manager.cc:1428] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:37.456895 339 tablet_bootstrap.cc:492] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.457715 32342 tablet_service.cc:1468] Processing CreateTablet for tablet c8e4cde2af694f07a9176313b4b93a46 (DEFAULT_TABLE table=default.Table_f933517ba72b4e44bd0432107e9a9f9f [id=8a7b2a641e2f4e638cef8b6604ad12f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.457252 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 171) took 3032 ms. Trace:
I20250623 08:04:37.458647 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 15154c85de7144a9b9e2367c70f57e2e (DEFAULT_TABLE table=default.Table_7c712a75a62f41dba90fcf504f4f752c [id=780a3bc4b3004e06a840a0b63e0c37c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.458118 979 rpcz_store.cc:276] 0623 08:04:34.424340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.424397 (+ 57us) service_pool.cc:225] Handling call
0623 08:04:37.400954 (+2976557us) ts_tablet_manager.cc:1145] Deleting tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0
0623 08:04:37.400965 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.432094 (+ 31129us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.452326 (+ 20232us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.457078 (+ 4752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2976192}
I20250623 08:04:37.459398 339 tablet_bootstrap.cc:654] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.460050 32483 tablet_service.cc:1515] Processing DeleteTablet for tablet 20e7a78cfc384033810603316b9e66bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.460254 32498 tablet_service.cc:1515] Processing DeleteTablet for tablet 136e482995a24c5daec55a701f3091fa with delete_type TABLET_DATA_DELETED (replaced by tablet 3bfad68d1f20486997ea1b0a8d97206b) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.460319 32495 tablet_service.cc:1468] Processing CreateTablet for tablet f0e24d8d890d4d1db051f172b3ec96ec (DEFAULT_TABLE table=default.Table_56c6dd0a0ee64ed088889c23db0fd2e9 [id=4199d682d171458ab79e4a36807c7f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.460669 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e24d8d890d4d1db051f172b3ec96ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.460780 32484 tablet_service.cc:1468] Processing CreateTablet for tablet c8e4cde2af694f07a9176313b4b93a46 (DEFAULT_TABLE table=default.Table_f933517ba72b4e44bd0432107e9a9f9f [id=8a7b2a641e2f4e638cef8b6604ad12f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.460937 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 283bebc7063d437ba3e541d7dee3c0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.461160 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet a34da529ffde4595a23fc6596b30e3e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cee3de5953244f98428747bea912f77) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.461153 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 19536c16ff32460e9db571e4ce411bab (table default.Table_f3f5e7c3d8e24a4aba9195d8de6c2030 [id=f8961c82f9384b0db128a791e00351f6]) successfully deleted
I20250623 08:04:37.461254 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e4cde2af694f07a9176313b4b93a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.450162 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 96) took 5844 ms. Trace:
I20250623 08:04:37.461361 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 8c01121f2e644d6aa6e8ec228bf75be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.461513 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b04e38d3da44301a450fe1fb5e2ada6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f416cdb596a4ed5a302acd07bb84b9c) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.461793 32487 tablet_service.cc:1468] Processing CreateTablet for tablet 207c75acfed64082a885b47ae5353c9d (DEFAULT_TABLE table=default.Table_c67b6294376343b38de7a2aa4ed7bcea [id=898fc58a25554c09867e70728f4a7508]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.462123 32487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 207c75acfed64082a885b47ae5353c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.463004 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 15154c85de7144a9b9e2367c70f57e2e (DEFAULT_TABLE table=default.Table_7c712a75a62f41dba90fcf504f4f752c [id=780a3bc4b3004e06a840a0b63e0c37c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.463297 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15154c85de7144a9b9e2367c70f57e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.463539 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 3fcd0382328f4b0f9ac510c83eb1f68f (DEFAULT_TABLE table=default.Table_e9f2ab1d95fb4341b5d05f6f5bd09f15 [id=f0a8652a629545a9a9bc4c22aba0da74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.463876 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fcd0382328f4b0f9ac510c83eb1f68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.464862 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 1e47b10f46b8402e94354ebca573c194 (table default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]) successfully deleted
I20250623 08:04:37.466423 32496 tablet_service.cc:1468] Processing CreateTablet for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (DEFAULT_TABLE table=default.Table_56d841d4028b4bcf9282392e22379235 [id=123fe4cae52542ddbde750a8de19320f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.466778 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eebc32fdfa44abcbd6f04a636d02c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.466996 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 30dfa986dfd44658a0ffb07796030520 (DEFAULT_TABLE table=default.Table_738cccce8e6c47d683b9a7c569aa58e3 [id=20baa502b37c4ec1904f52ddeaf99c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.467305 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30dfa986dfd44658a0ffb07796030520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.469926 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 5fdc9e53da264a5ca1f70e1846c906f0 (table default.Table_0242542a86ba40a28474c9832cd8ed3c [id=4e8632de2b4744ff9de198f82720a5bc]) successfully deleted
I20250623 08:04:37.470678 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet f3d32798168345edb9898751d083057c (table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]) successfully deleted
I20250623 08:04:37.471118 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 492ffaeaaa314dcfb7512d8f23946a67 (DEFAULT_TABLE table=default.Table_712680c326284dd782d57dc0ee153879 [id=407f4156ebee45ecb729ad3b3d5f26e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.471122 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecb422550efd40cf82bc92fca5c25eda" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.471447 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 492ffaeaaa314dcfb7512d8f23946a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.472133 32376 raft_consensus.cc:3058] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.472661 32377 raft_consensus.cc:3058] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.473685 342 tablet_bootstrap.cc:654] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.474817 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d98654c29b9c4e75af6a213cafd4c2c9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:37.476178 32498 tablet_service.cc:1468] Processing CreateTablet for tablet dcfe68d4e6ae470d816ac4c494dcaa51 (DEFAULT_TABLE table=default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe [id=f77e622d3eb44f4fa3a284b08eefc109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.476516 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcfe68d4e6ae470d816ac4c494dcaa51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.476639 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fd478c007914d2cb09d929b50322a64" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.479108 32483 tablet_service.cc:1468] Processing CreateTablet for tablet 36d21fe528f5410eacd655bf15306b98 (DEFAULT_TABLE table=default.Table_a50378b31cda420a8791443844eed5f2 [id=b83148d602ab4c19a5fb1b4072b626e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.479465 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36d21fe528f5410eacd655bf15306b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.481207 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eebc32fdfa44abcbd6f04a636d02c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.481360 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcfe68d4e6ae470d816ac4c494dcaa51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.481738 32349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30dfa986dfd44658a0ffb07796030520. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.482127 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fcd0382328f4b0f9ac510c83eb1f68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.482298 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e24d8d890d4d1db051f172b3ec96ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.482470 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 207c75acfed64082a885b47ae5353c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.482933 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 492ffaeaaa314dcfb7512d8f23946a67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.483091 32375 raft_consensus.cc:3058] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.483319 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36d21fe528f5410eacd655bf15306b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.483811 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15154c85de7144a9b9e2367c70f57e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.486397 32342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e4cde2af694f07a9176313b4b93a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.486919 1303 raft_consensus.cc:491] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.487043 1303 raft_consensus.cc:513] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.487632 1303 leader_election.cc:290] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.489804 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 771ecfc41e2b4007a4d213cec5b128c6 with delete_type TABLET_DATA_DELETED (replaced by tablet e41b15c89a744a698ca5eee2f143b4bc) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:37.503319 32453 leader_election.cc:343] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 75a6ebe0d6b0463b8ddcb1a091a2cc6f
I20250623 08:04:37.503703 32453 leader_election.cc:304] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.505899 1359 raft_consensus.cc:2747] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:37.506646 1359 consensus_queue.cc:1035] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.508553 1359 consensus_queue.cc:1035] T 0c58231f67604422a1497e4c1b44ec52 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.509877 32453 leader_election.cc:304] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.510352 1359 raft_consensus.cc:2802] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:37.510555 1359 raft_consensus.cc:695] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:37.510792 1359 consensus_queue.cc:237] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.516891 342 tablet_bootstrap.cc:492] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.517077 342 ts_tablet_manager.cc:1397] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.000s
I20250623 08:04:37.517797 342 raft_consensus.cc:357] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.518007 342 raft_consensus.cc:383] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.518093 342 raft_consensus.cc:738] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.518321 342 consensus_queue.cc:260] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.518858 342 ts_tablet_manager.cc:1428] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:37.519151 342 tablet_bootstrap.cc:492] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.520486 342 tablet_bootstrap.cc:654] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.522486 339 tablet_bootstrap.cc:492] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.522670 339 ts_tablet_manager.cc:1397] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.000s
I20250623 08:04:37.523398 339 raft_consensus.cc:357] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.523599 339 raft_consensus.cc:383] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.523691 339 raft_consensus.cc:738] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.523912 339 consensus_queue.cc:260] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.524381 339 ts_tablet_manager.cc:1428] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:37.525689 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.526284 339 tablet_bootstrap.cc:492] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.528201 339 tablet_bootstrap.cc:654] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.531266 1575 consensus_queue.cc:1035] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.532130 1575 consensus_queue.cc:1035] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.534937 1575 consensus_queue.cc:1035] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.536144 1575 raft_consensus.cc:491] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.536257 1575 raft_consensus.cc:513] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.536998 1575 leader_election.cc:290] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.461443 952 rpcz_store.cc:276] 0623 08:04:31.605987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.608847 (+ 2860us) service_pool.cc:225] Handling call
0623 08:04:37.425989 (+5817142us) ts_tablet_manager.cc:1145] Deleting tablet 38ecd4c243b840f4b66d8e01078dca64
0623 08:04:37.425998 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.432522 (+ 6524us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.448477 (+ 15955us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.450110 (+ 1633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5792643}
I20250623 08:04:37.540354 952 tablet_replica.cc:331] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.540707 952 raft_consensus.cc:2241] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.541030 952 raft_consensus.cc:2270] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.541085 339 tablet_bootstrap.cc:492] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.541312 981 ts_tablet_manager.cc:1918] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:37.541472 981 log.cc:1199] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/a25bdc9b7e48467fba3771365e58bcbc
I20250623 08:04:37.541848 981 ts_tablet_manager.cc:1939] T a25bdc9b7e48467fba3771365e58bcbc P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:37.542110 952 ts_tablet_manager.cc:1905] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.542163 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 161) took 2898 ms. Trace:
I20250623 08:04:37.542255 981 rpcz_store.cc:276] 0623 08:04:34.644196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.645338 (+ 1142us) service_pool.cc:225] Handling call
0623 08:04:37.194018 (+2548680us) ts_tablet_manager.cc:1145] Deleting tablet a25bdc9b7e48467fba3771365e58bcbc
0623 08:04:37.194031 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.303327 (+109296us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.541209 (+237882us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.542146 (+ 937us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2548444}
I20250623 08:04:37.542709 981 tablet_replica.cc:331] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:37.542656 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]) successfully deleted
I20250623 08:04:37.542908 981 raft_consensus.cc:2241] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.543059 981 raft_consensus.cc:2270] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:37.549355 32044 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 23928d64bcef4c06af81698e688b7f94 consensus state has no leader
I20250623 08:04:37.549331 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecb422550efd40cf82bc92fca5c25eda" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.549541 32646 raft_consensus.cc:2466] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:37.549618 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 172) took 3123 ms. Trace:
I20250623 08:04:37.547034 32626 tablet_service.cc:1468] Processing CreateTablet for tablet dcfe68d4e6ae470d816ac4c494dcaa51 (DEFAULT_TABLE table=default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe [id=f77e622d3eb44f4fa3a284b08eefc109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.549721 979 rpcz_store.cc:276] 0623 08:04:34.426004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.426066 (+ 62us) service_pool.cc:225] Handling call
0623 08:04:37.549480 (+3123414us) ts_tablet_manager.cc:1145] Deleting tablet 92005abc61dd40ef9da8124cadc29b6a
0623 08:04:37.549490 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.549594 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3123137}
I20250623 08:04:37.550055 979 tablet_replica.cc:331] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:37.550112 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcfe68d4e6ae470d816ac4c494dcaa51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.547216 32627 tablet_service.cc:1468] Processing CreateTablet for tablet c8e4cde2af694f07a9176313b4b93a46 (DEFAULT_TABLE table=default.Table_f933517ba72b4e44bd0432107e9a9f9f [id=8a7b2a641e2f4e638cef8b6604ad12f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.548197 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d98654c29b9c4e75af6a213cafd4c2c9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:37.550504 32653 raft_consensus.cc:3058] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.548768 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 15154c85de7144a9b9e2367c70f57e2e (DEFAULT_TABLE table=default.Table_7c712a75a62f41dba90fcf504f4f752c [id=780a3bc4b3004e06a840a0b63e0c37c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.551249 981 ts_tablet_manager.cc:1905] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.541347 339 ts_tablet_manager.cc:1397] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250623 08:04:37.557426 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15154c85de7144a9b9e2367c70f57e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.557934 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.558099 32646 raft_consensus.cc:2466] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:37.558110 339 raft_consensus.cc:357] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.558248 339 raft_consensus.cc:383] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.558470 32314 leader_election.cc:304] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:37.558971 1303 raft_consensus.cc:2802] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:37.559044 1303 raft_consensus.cc:491] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.559103 1303 raft_consensus.cc:3058] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:37.543638 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:37.561703 1303 raft_consensus.cc:513] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.558301 339 raft_consensus.cc:738] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.549042 1507 raft_consensus.cc:491] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.562688 339 consensus_queue.cc:260] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.562650 1507 raft_consensus.cc:513] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.563269 1507 leader_election.cc:290] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.563383 339 ts_tablet_manager.cc:1428] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250623 08:04:37.563584 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:37.563656 339 tablet_bootstrap.cc:492] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.562295 1303 leader_election.cc:290] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.564554 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7d69bcad8684d99a0b08eefb470ab01" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:37.564695 32650 raft_consensus.cc:3058] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:37.566757 979 raft_consensus.cc:2241] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:37.567111 979 raft_consensus.cc:2270] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.567821 339 tablet_bootstrap.cc:654] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.567793 979 ts_tablet_manager.cc:1905] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.568439 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 (table default.Table_3ef2e86e99164cce8e882e6954e2ec17 [id=dbbd669a72954d1aaf169b5870a85fc3]) successfully deleted
W20250623 08:04:37.568761 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 92005abc61dd40ef9da8124cadc29b6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92005abc61dd40ef9da8124cadc29b6a already in progress: creating tablet
I20250623 08:04:37.577540 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:37.577943 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e4cde2af694f07a9176313b4b93a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.577967 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]). Expected version 2 got 0
I20250623 08:04:37.578133 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:37.583365 981 ts_tablet_manager.cc:1918] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.583473 981 log.cc:1199] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/8ffafd10f26740c9a5b6c9bdc3bb23a0
I20250623 08:04:37.583814 981 ts_tablet_manager.cc:1939] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:37.584100 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 162) took 2938 ms. Trace:
I20250623 08:04:37.584165 981 rpcz_store.cc:276] 0623 08:04:34.645823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.646177 (+ 354us) service_pool.cc:225] Handling call
0623 08:04:37.542662 (+2896485us) ts_tablet_manager.cc:1145] Deleting tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0
0623 08:04:37.542672 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.571992 (+ 29320us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.583260 (+ 11268us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.584085 (+ 825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896280}
I20250623 08:04:37.584668 981 tablet_replica.cc:331] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:37.584860 981 raft_consensus.cc:2241] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.585012 981 raft_consensus.cc:2270] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.588459 1575 raft_consensus.cc:491] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.588582 1575 raft_consensus.cc:513] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.589092 1575 leader_election.cc:290] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.589102 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 90ceb0f8d5c94c4e9427042d8a35e16c (table default.Table_0972bb6b77644e27b56f3e0007bc5fd4 [id=6fa96454486845848e54e86fd64a0017]). Expected version 1 got 0
I20250623 08:04:37.589452 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 (table default.Table_3ef2e86e99164cce8e882e6954e2ec17 [id=dbbd669a72954d1aaf169b5870a85fc3]) successfully deleted
I20250623 08:04:37.590785 981 ts_tablet_manager.cc:1905] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.589427 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:37.591204 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:37.591758 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:37.593881 1440 raft_consensus.cc:491] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.595043 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:37.595546 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:37.595968 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fd478c007914d2cb09d929b50322a64" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.596084 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 59015204545e436cb558fe4465a67c97 (table default.Table_094f2e2ebe104774b6a713a5bbb7fd2f [id=9979c6d8878443c09e6ff9f4aea65d62]). Expected version 1 got 0
I20250623 08:04:37.596829 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0db9480bb07343fe935fe9bfc98bca77 (table default.Table_1f98d716ca3541379adbced08214bfdd [id=d1fbcd8e89164d71b021f4b1f4223a59]). Expected version 1 got 0
I20250623 08:04:37.597153 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74842fde5152419ca35f3950e465247a (table default.Table_9aeb558e2ae64fe2b45bef5d2b6a2625 [id=2bfacaaab9f347b1a77e5eb29a9c485e]). Expected version 1 got 0
I20250623 08:04:37.597766 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:37.598472 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]). Expected version 1 got 0
I20250623 08:04:37.594007 1440 raft_consensus.cc:513] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.599237 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_83732a7d68bb412f9c26b799fc16e4ec [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 1 got 0
I20250623 08:04:37.596138 32646 raft_consensus.cc:2466] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:37.599684 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_bc1251ce708a4467b5b1c71a1df307e0 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 1 got 0
I20250623 08:04:37.599871 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:37.600101 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet d5cd316e6f4447be958611ab07ed31fc (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:37.603652 1440 leader_election.cc:290] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:37.603785 1440 raft_consensus.cc:491] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.603878 1440 raft_consensus.cc:513] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.604373 1440 leader_election.cc:290] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:37.606663 32650 raft_consensus.cc:2466] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 2.
I20250623 08:04:37.607404 32314 leader_election.cc:304] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:37.607714 1303 raft_consensus.cc:2802] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:37.607954 1303 raft_consensus.cc:695] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:37.608229 1303 consensus_queue.cc:237] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.611218 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:37.613323 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:37.614080 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:37.607096 32649 raft_consensus.cc:1238] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Rejecting Update request from peer 59f7c614254f42009306b13337f06576 for earlier term 1. Current term is 2. Ops: []
I20250623 08:04:37.614411 32653 raft_consensus.cc:2466] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:37.615222 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:37.615593 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 025365d2fa0a43dd983d395a05cc2af1 (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:37.616112 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:37.613888 952 ts_tablet_manager.cc:1918] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:37.616776 952 log.cc:1199] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/70f9d21b8c144f9cada9bdbe0fbb1538
I20250623 08:04:37.617409 981 ts_tablet_manager.cc:1918] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.617967 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:37.618960 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:37.619843 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:37.619607 952 ts_tablet_manager.cc:1939] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:37.620235 981 log.cc:1199] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/f3d32798168345edb9898751d083057c
I20250623 08:04:37.622570 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 95) took 6016 ms. Trace:
I20250623 08:04:37.623274 981 ts_tablet_manager.cc:1939] T f3d32798168345edb9898751d083057c P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:37.623622 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 163) took 2977 ms. Trace:
I20250623 08:04:37.624634 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_aaa3ca8d9bb0406981f0dfff8cbf9e3c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 4 got 0
I20250623 08:04:37.623432 952 rpcz_store.cc:276] 0623 08:04:31.605829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.608734 (+ 2905us) service_pool.cc:225] Handling call
0623 08:04:37.540278 (+5931544us) ts_tablet_manager.cc:1145] Deleting tablet 70f9d21b8c144f9cada9bdbe0fbb1538
0623 08:04:37.540287 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.562283 (+ 21996us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.612858 (+ 50575us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.622544 (+ 9686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5894128}
I20250623 08:04:37.625525 981 rpcz_store.cc:276] 0623 08:04:34.646657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.660020 (+ 13363us) service_pool.cc:225] Handling call
0623 08:04:37.584614 (+2924594us) ts_tablet_manager.cc:1145] Deleting tablet f3d32798168345edb9898751d083057c
0623 08:04:37.584623 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.601747 (+ 17124us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.617282 (+ 15535us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.623602 (+ 6320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2924364}
I20250623 08:04:37.626078 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:37.626022 952 tablet_replica.cc:331] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.626915 981 tablet_replica.cc:331] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:37.626667 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet f3d32798168345edb9898751d083057c (table default.Table_4ee0e165259b4e2c88626b2d613149ca [id=5a4c53ad062644a49ea9c8d3ca32e79a]) successfully deleted
I20250623 08:04:37.627657 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:37.627970 981 raft_consensus.cc:2241] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.628409 981 raft_consensus.cc:2270] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.629017 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:37.629590 981 ts_tablet_manager.cc:1905] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.629179 952 raft_consensus.cc:2241] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.630232 952 raft_consensus.cc:2270] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.630643 32050 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:37.632934 952 ts_tablet_manager.cc:1905] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.633744 343 tablet_bootstrap.cc:654] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.635712 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet c0773fc60d4a463596f536f15d58bf57
I20250623 08:04:37.636162 979 ts_tablet_manager.cc:1918] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.636389 979 log.cc:1199] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/cdec6e9573b041e0a57d1e0e0239d97f
I20250623 08:04:37.636557 32058 master_service.cc:948] ReplaceTablet: received request to replace tablet f279b63b55c64e548ded27b02025c9f5 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:37.637279 979 ts_tablet_manager.cc:1939] T cdec6e9573b041e0a57d1e0e0239d97f P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.637866 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 174) took 3186 ms. Trace:
I20250623 08:04:37.637935 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet c0773fc60d4a463596f536f15d58bf57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.638006 979 rpcz_store.cc:276] 0623 08:04:34.451115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.451557 (+ 442us) service_pool.cc:225] Handling call
0623 08:04:37.550005 (+3098448us) ts_tablet_manager.cc:1145] Deleting tablet cdec6e9573b041e0a57d1e0e0239d97f
0623 08:04:37.550016 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.588501 (+ 38485us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.629119 (+ 40618us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.637821 (+ 8702us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16413,"tablet-delete.queue_time_us":3098160}
I20250623 08:04:37.638761 32059 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_9aeb558e2ae64fe2b45bef5d2b6a2625 [id=2bfacaaab9f347b1a77e5eb29a9c485e] alter complete (version 1)
W20250623 08:04:37.639261 32055 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 077bb090b3ea4d029463304d7a6feee5 consensus state has no leader
I20250623 08:04:37.639693 32055 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_dd7b43110df2433597689f77b73907a1" } modify_external_catalogs: true
I20250623 08:04:37.639747 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet cdec6e9573b041e0a57d1e0e0239d97f (table default.Table_48cbdc58fd034e6390087ed7c6346d24 [id=ef1a8122b8044388b4ae24a03da26837]) successfully deleted
I20250623 08:04:37.639940 32055 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_dd7b43110df2433597689f77b73907a1" } modify_external_catalogs: true
W20250623 08:04:37.640650 32059 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:39129 (request call id 174) took 1057 ms (client timeout 1000 ms). Trace:
W20250623 08:04:37.641165 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0d1ec9d56ba40128325a50f204fe920 consensus state has no leader
W20250623 08:04:37.640744 32059 rpcz_store.cc:269] 0623 08:04:36.582904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.618356 (+ 35452us) service_pool.cc:225] Handling call
0623 08:04:37.640578 (+1022222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:37.253177 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:37.253280 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:37.253286 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:37.253289 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:37.253380 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:37.253392 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:37.253397 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:37.253452 (+ 55us) tablet.cc:699] Row locks acquired
0623 08:04:37.253455 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:37.253485 (+ 30us) write_op.cc:270] Start()
0623 08:04:37.253515 (+ 30us) write_op.cc:276] Timestamp: P: 1750665877253422 usec, L: 0
0623 08:04:37.253520 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:37.253711 (+ 191us) log.cc:844] Serialized 356 byte log entry
0623 08:04:37.256218 (+ 2507us) op_driver.cc:464] REPLICATION: finished
0623 08:04:37.256324 (+ 106us) write_op.cc:301] APPLY: starting
0623 08:04:37.256337 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:37.256373 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:37.256401 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:37.256444 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:37.256474 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:37.256482 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:37.256538 (+ 56us) log.cc:844] Serialized 23 byte log entry
0623 08:04:37.256586 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:37.256601 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:37.256606 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:37.256676 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:37.256680 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":344,"apply.run_wall_time_us":345,"num_ops":1,"prepare.queue_time_us":107,"prepare.run_cpu_time_us":621,"prepare.run_wall_time_us":622,"raft.queue_time_us":60,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":215,"replication_time_us":2672}]]}
I20250623 08:04:37.641209 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet c0773fc60d4a463596f536f15d58bf57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:37.641209 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 175) took 3187 ms. Trace:
I20250623 08:04:37.641534 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_37287a2bae6a4d72ab9a957a92be554b" } new_table_name: "default.Table_06be929600454320935c1f072c6629dc" modify_external_catalogs: true
I20250623 08:04:37.641639 32059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_62ddca96ee344b168687745f9ef92af4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.642197 32059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62ddca96ee344b168687745f9ef92af4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.641661 979 rpcz_store.cc:276] 0623 08:04:34.453536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.454777 (+ 1241us) service_pool.cc:225] Handling call
0623 08:04:37.640526 (+3185749us) ts_tablet_manager.cc:1145] Deleting tablet 966a159e212040109e2842c15b69fb71
0623 08:04:37.640695 (+ 169us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.641087 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3180106}
I20250623 08:04:37.643275 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 176) took 3179 ms. Trace:
W20250623 08:04:37.643795 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 966a159e212040109e2842c15b69fb71 because the tablet was not found. No further retry: Not found: Tablet not found: 966a159e212040109e2842c15b69fb71
I20250623 08:04:37.643965 979 rpcz_store.cc:276] 0623 08:04:34.463825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.463884 (+ 59us) service_pool.cc:225] Handling call
0623 08:04:37.642868 (+3178984us) ts_tablet_manager.cc:1145] Deleting tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa
0623 08:04:37.642877 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.643259 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3172713}
I20250623 08:04:37.646688 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet c7fcf583064f4538913d9157e93068dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:37.654172 32059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_25f75684255a4b99a8aff38ee935aac5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.654716 32059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25f75684255a4b99a8aff38ee935aac5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:37.659130 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 183202us
I20250623 08:04:37.659248 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet cdec6e9573b041e0a57d1e0e0239d97f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:37.660033 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52d0a356c74f4daea9093c998b4bf73e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:37.660769 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa because the tablet was not found. No further retry: Not found: Tablet not found: 9bc95f12ab5d4b9b9f0729cb88aabbaa
I20250623 08:04:37.660914 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 177) took 3196 ms. Trace:
I20250623 08:04:37.661512 979 rpcz_store.cc:276] 0623 08:04:34.464401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.464448 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:37.660278 (+3195830us) ts_tablet_manager.cc:1145] Deleting tablet ea702abb31fb41498ab61bbb05b1f7e2
0623 08:04:37.660293 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.660890 (+ 597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3190023}
I20250623 08:04:37.663273 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f76ef24c69f4b17a373b242d71da8b9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:37.663304 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet ea702abb31fb41498ab61bbb05b1f7e2 because the tablet was not found. No further retry: Not found: Tablet not found: ea702abb31fb41498ab61bbb05b1f7e2
I20250623 08:04:37.663559 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f9f4829e103b460eb582beff3b966bb4
I20250623 08:04:37.664404 1506 raft_consensus.cc:491] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.664335 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_a935cce17e794f328640dde460d22086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:37.664517 1506 raft_consensus.cc:513] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.664775 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet f9f4829e103b460eb582beff3b966bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:37.664852 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a935cce17e794f328640dde460d22086 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.665031 1506 leader_election.cc:290] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:37.665941 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet f9f4829e103b460eb582beff3b966bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.665985 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 178) took 3201 ms. Trace:
I20250623 08:04:37.666993 979 rpcz_store.cc:276] 0623 08:04:34.464873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.464932 (+ 59us) service_pool.cc:225] Handling call
0623 08:04:37.665369 (+3200437us) ts_tablet_manager.cc:1145] Deleting tablet f0117cbb09dd4067836266c9d6a32adf
0623 08:04:37.665555 (+ 186us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.665963 (+ 408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3194929}
W20250623 08:04:37.668874 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f0117cbb09dd4067836266c9d6a32adf because the tablet was not found. No further retry: Not found: Tablet not found: f0117cbb09dd4067836266c9d6a32adf
I20250623 08:04:37.669840 32059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_a8ada1b74f854fe58c88fc3d90167bd3" } new_table_name: "default.Table_dab86e75c8674afab09def2e63dbfb43" modify_external_catalogs: true
I20250623 08:04:37.670332 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 179) took 3205 ms. Trace:
I20250623 08:04:37.672022 32568 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:37.671048 979 rpcz_store.cc:276] 0623 08:04:34.465330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.465394 (+ 64us) service_pool.cc:225] Handling call
0623 08:04:37.669727 (+3204333us) ts_tablet_manager.cc:1145] Deleting tablet 79731106915d4cebb9dcc3c10386ffa9
0623 08:04:37.669741 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.670146 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3199316}
W20250623 08:04:37.673362 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 79731106915d4cebb9dcc3c10386ffa9 because the tablet was not found. No further retry: Not found: Tablet not found: 79731106915d4cebb9dcc3c10386ffa9
I20250623 08:04:37.675257 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 180) took 3209 ms. Trace:
I20250623 08:04:37.675786 979 rpcz_store.cc:276] 0623 08:04:34.465767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.465814 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:37.674692 (+3208878us) ts_tablet_manager.cc:1145] Deleting tablet c6a97b34e386443fa19a9601621b5017
0623 08:04:37.674704 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.675240 (+ 536us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3204184}
I20250623 08:04:37.678772 979 tablet_replica.cc:331] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:37.680567 979 raft_consensus.cc:2241] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:37.681527 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet c6a97b34e386443fa19a9601621b5017 because the tablet was not found. No further retry: Not found: Tablet not found: c6a97b34e386443fa19a9601621b5017
I20250623 08:04:37.682155 979 raft_consensus.cc:2270] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.686225 1332 raft_consensus.cc:491] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.686374 1332 raft_consensus.cc:513] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.687151 1332 leader_election.cc:290] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.687624 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b19e5f2359a84c7ea02a0b25240a54cb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.687702 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b19e5f2359a84c7ea02a0b25240a54cb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.687945 32648 raft_consensus.cc:2391] T b19e5f2359a84c7ea02a0b25240a54cb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
W20250623 08:04:37.688058 32059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 297bdf6c0e424190a35b55cbeefe7161 consensus state has no leader
I20250623 08:04:37.688333 32314 leader_election.cc:304] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:37.688616 1332 raft_consensus.cc:3058] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.688972 979 ts_tablet_manager.cc:1905] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.692160 1332 raft_consensus.cc:2747] T b19e5f2359a84c7ea02a0b25240a54cb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:37.666500 32589 leader_election.cc:343] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.693612 32589 leader_election.cc:304] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.694039 1506 raft_consensus.cc:2747] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:37.694715 32589 leader_election.cc:304] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.695001 1506 raft_consensus.cc:2802] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:37.695084 1506 raft_consensus.cc:491] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.695163 1506 raft_consensus.cc:3058] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.699864 1506 raft_consensus.cc:513] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.708066 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7
I20250623 08:04:37.709790 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:37.709860 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:37.710204 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f83f68ebda7e4826bbb17278cd4ac5a8
I20250623 08:04:37.711040 1506 leader_election.cc:290] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:37.711783 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet f83f68ebda7e4826bbb17278cd4ac5a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:37.711815 1332 raft_consensus.cc:491] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.711938 1332 raft_consensus.cc:513] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.712021 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet f83f68ebda7e4826bbb17278cd4ac5a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:35542
W20250623 08:04:37.712489 32052 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 736eb56dae194df2af90bf33d3294a4e consensus state has no leader
I20250623 08:04:37.712556 1332 leader_election.cc:290] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:37.718030 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd7f3d2e67e84bfcbd45a21658f69e7b consensus state has no leader
I20250623 08:04:37.718730 952 ts_tablet_manager.cc:1918] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:37.719871 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a414144276f6484ea4690204752e2142" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.720220 952 log.cc:1199] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/75a6ebe0d6b0463b8ddcb1a091a2cc6f
I20250623 08:04:37.722416 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a414144276f6484ea4690204752e2142" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.723308 32314 leader_election.cc:304] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:37.723588 1332 raft_consensus.cc:2747] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:37.723932 952 ts_tablet_manager.cc:1939] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:37.725922 32056 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:39501 (request call id 169) took 1708 ms (client timeout 1000 ms). Trace:
W20250623 08:04:37.726199 32056 rpcz_store.cc:269] 0623 08:04:36.017048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.024298 (+ 7250us) service_pool.cc:225] Handling call
0623 08:04:37.725885 (+1701587us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:36.431780 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:36.431891 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:36.431896 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:36.431898 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:36.432089 (+ 191us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:36.432097 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:36.432102 (+ 5us) tablet.cc:683] Acquiring locks for 5 operations
0623 08:04:36.432270 (+ 168us) tablet.cc:699] Row locks acquired
0623 08:04:36.432274 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:36.432306 (+ 32us) write_op.cc:270] Start()
0623 08:04:36.432330 (+ 24us) write_op.cc:276] Timestamp: P: 1750665876432274 usec, L: 0
0623 08:04:36.432333 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:36.434522 (+ 2189us) log.cc:844] Serialized 2024 byte log entry
0623 08:04:36.497446 (+ 62924us) op_driver.cc:464] REPLICATION: finished
0623 08:04:36.498118 (+ 672us) write_op.cc:301] APPLY: starting
0623 08:04:36.498129 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:36.498241 (+ 112us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:36.498275 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:36.498397 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:36.498426 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0623 08:04:36.498438 (+ 12us) write_op.cc:312] APPLY: finished
0623 08:04:36.498500 (+ 62us) log.cc:844] Serialized 47 byte log entry
0623 08:04:36.498537 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:36.498554 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:36.498559 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:36.498629 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:36.498633 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8839,"reported_tablets":171,"child_traces":[["op",{"apply.queue_time_us":643,"apply.run_cpu_time_us":494,"apply.run_wall_time_us":493,"num_ops":5,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":778,"prepare.run_wall_time_us":2801,"replication_time_us":65095}]]}
I20250623 08:04:37.726836 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 97) took 6120 ms. Trace:
I20250623 08:04:37.728132 32056 master_service.cc:948] ReplaceTablet: received request to replace tablet 417a050eada4403ebbddff255fb6fb5a from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:37.727705 952 rpcz_store.cc:276] 0623 08:04:31.606089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.629821 (+ 23732us) service_pool.cc:225] Handling call
0623 08:04:37.625647 (+5995826us) ts_tablet_manager.cc:1145] Deleting tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f
0623 08:04:37.625663 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.681818 (+ 56155us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.717872 (+ 36054us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.726429 (+ 8557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5942454}
I20250623 08:04:37.731321 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 98) took 5870 ms. Trace:
W20250623 08:04:37.732263 32453 leader_election.cc:343] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.732394 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.732556 32648 raft_consensus.cc:2466] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:37.733081 1507 consensus_queue.cc:1035] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:37.733256 32453 leader_election.cc:343] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.733569 1507 consensus_queue.cc:1035] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.733860 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.732331 952 rpcz_store.cc:276] 0623 08:04:31.860477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.862789 (+ 2312us) service_pool.cc:225] Handling call
0623 08:04:37.730321 (+5867532us) ts_tablet_manager.cc:1145] Deleting tablet 6926c49b787941d98c647543bd3b9ed5
0623 08:04:37.730333 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.731302 (+ 969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5666551}
I20250623 08:04:37.734462 952 tablet_replica.cc:331] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.734650 952 raft_consensus.cc:2241] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.737962 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_54fe37774d62482296afe3191f2e8e88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.738528 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54fe37774d62482296afe3191f2e8e88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.740929 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_412c3eb097714d94bf0b6c487f1c3caa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.741415 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_412c3eb097714d94bf0b6c487f1c3caa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.743021 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_f883c19d25584eb185a1e31ef6d88d4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.743108 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 48415.1us
W20250623 08:04:37.743454 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f883c19d25584eb185a1e31ef6d88d4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.744760 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_f663b3b4536b4a19a3c42d533d148127"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:37.745110 979 ts_tablet_manager.cc:1918] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:37.745208 979 log.cc:1199] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/6c6f98e2db1a4b0789c5819b2c7d4bb6
W20250623 08:04:37.745179 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f663b3b4536b4a19a3c42d533d148127 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.745497 979 ts_tablet_manager.cc:1939] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.745806 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 194) took 2858 ms. Trace:
I20250623 08:04:37.745877 979 rpcz_store.cc:276] 0623 08:04:34.887333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.887386 (+ 53us) service_pool.cc:225] Handling call
0623 08:04:37.678259 (+2790873us) ts_tablet_manager.cc:1145] Deleting tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6
0623 08:04:37.678273 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.720616 (+ 42343us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.745004 (+ 24388us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.745791 (+ 787us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2790420}
I20250623 08:04:37.746243 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 195) took 2642 ms. Trace:
I20250623 08:04:37.742622 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_ddd07329c08c4b6e8468335182c7c7e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:37.746312 979 rpcz_store.cc:276] 0623 08:04:35.103505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.103579 (+ 74us) service_pool.cc:225] Handling call
0623 08:04:37.746130 (+2642551us) ts_tablet_manager.cc:1145] Deleting tablet 36c8714cfbd84dd38cf5c0f305ef6843
0623 08:04:37.746138 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.746229 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2642302}
I20250623 08:04:37.746567 979 tablet_replica.cc:331] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
W20250623 08:04:37.746790 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddd07329c08c4b6e8468335182c7c7e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.746805 952 raft_consensus.cc:2270] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.747807 952 ts_tablet_manager.cc:1905] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:37.733626 32453 consensus_peers.cc:487] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:37.748415 32453 leader_election.cc:343] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.749272 1447 consensus_queue.cc:1035] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:37.751513 32453 leader_election.cc:343] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.753626 32059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_afe210848a6b49b2a15d04cf79e1d47c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.754132 32059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afe210848a6b49b2a15d04cf79e1d47c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.755434 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 38ecd4c243b840f4b66d8e01078dca64 (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]) successfully deleted
I20250623 08:04:37.755690 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]) successfully deleted
W20250623 08:04:37.755932 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 36c8714cfbd84dd38cf5c0f305ef6843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36c8714cfbd84dd38cf5c0f305ef6843 already in progress: creating tablet
I20250623 08:04:37.756238 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52d0a356c74f4daea9093c998b4bf73e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.757658 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21080ab069d74457a9f7fe3aabdb3c7c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.759446 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 70f9d21b8c144f9cada9bdbe0fbb1538 (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]) successfully deleted
I20250623 08:04:37.746737 979 raft_consensus.cc:2241] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.761070 979 raft_consensus.cc:2270] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:37.752676 32589 leader_election.cc:343] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.764130 32589 leader_election.cc:304] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.753625 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52d0a356c74f4daea9093c998b4bf73e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.764400 1506 raft_consensus.cc:2747] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:37.764501 32649 raft_consensus.cc:2466] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:37.758447 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21080ab069d74457a9f7fe3aabdb3c7c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.764896 32646 raft_consensus.cc:2466] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:37.765774 343 tablet_bootstrap.cc:492] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:37.768738 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736eb56dae194df2af90bf33d3294a4e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:37.775498 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.775571 1332 consensus_queue.cc:1035] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.775769 32381 raft_consensus.cc:1238] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Rejecting Update request from peer 59f7c614254f42009306b13337f06576 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250623 08:04:37.777702 1507 raft_consensus.cc:491] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.777870 1507 raft_consensus.cc:513] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.778537 1507 leader_election.cc:290] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.783814 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52d0a356c74f4daea9093c998b4bf73e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.784088 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]) successfully deleted
W20250623 08:04:37.784608 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 6926c49b787941d98c647543bd3b9ed5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6926c49b787941d98c647543bd3b9ed5 already in progress: creating tablet
I20250623 08:04:37.784603 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f76ef24c69f4b17a373b242d71da8b9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.787559 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.787756 32513 raft_consensus.cc:2466] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:37.790330 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736eb56dae194df2af90bf33d3294a4e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:37.790539 32513 raft_consensus.cc:3058] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.791560 342 tablet_bootstrap.cc:492] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:37.791714 342 ts_tablet_manager.cc:1397] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.273s user 0.064s sys 0.064s
I20250623 08:04:37.792610 342 raft_consensus.cc:357] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.792812 342 raft_consensus.cc:383] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.792873 342 raft_consensus.cc:738] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.793530 32513 raft_consensus.cc:2466] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:37.798511 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet c0773fc60d4a463596f536f15d58bf57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.798606 32053 master_service.cc:948] ReplaceTablet: received request to replace tablet 025365d2fa0a43dd983d395a05cc2af1 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:37.799924 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet f9f4829e103b460eb582beff3b966bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.770484 979 ts_tablet_manager.cc:1905] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.800117 343 ts_tablet_manager.cc:1397] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.357s user 0.004s sys 0.008s
I20250623 08:04:37.800297 32057 master_service.cc:948] ReplaceTablet: received request to replace tablet 886fc0fa39c54bddb25ac01f68d95805 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:37.800877 343 raft_consensus.cc:357] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.801030 343 raft_consensus.cc:383] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.801077 343 raft_consensus.cc:738] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.801272 343 consensus_queue.cc:260] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.801816 343 ts_tablet_manager.cc:1428] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:37.802057 343 tablet_bootstrap.cc:492] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:37.803387 343 tablet_bootstrap.cc:654] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.806501 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:37.797042 32589 leader_election.cc:343] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.807832 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet f83f68ebda7e4826bbb17278cd4ac5a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:37.808089 32589 leader_election.cc:343] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.809145 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet cde7df7d39a54114a4dcb28cfd1eb429 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f1fb8231e8e45208ac4f412134d8696) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.810300 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet f0117cbb09dd4067836266c9d6a32adf with delete_type TABLET_DATA_DELETED (replaced by tablet 2ecf4af9e476495683bd63cac20d4672) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:37.809935 32589 leader_election.cc:343] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.811596 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa with delete_type TABLET_DATA_DELETED (replaced by tablet 6f9e0d1b85a94ae9ae86e53688a55db5) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:37.812029 32589 leader_election.cc:343] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.814184 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 19536c16ff32460e9db571e4ce411bab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.814980 342 consensus_queue.cc:260] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.815174 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 417a050eada4403ebbddff255fb6fb5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.815469 342 ts_tablet_manager.cc:1428] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250623 08:04:37.815910 342 tablet_bootstrap.cc:492] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:37.816341 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9e53da264a5ca1f70e1846c906f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.817546 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet f3d32798168345edb9898751d083057c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.819941 342 tablet_bootstrap.cc:654] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.821491 339 tablet_bootstrap.cc:492] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.821616 339 ts_tablet_manager.cc:1397] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.258s user 0.012s sys 0.006s
I20250623 08:04:37.822420 339 raft_consensus.cc:357] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.822573 339 raft_consensus.cc:383] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.822615 339 raft_consensus.cc:738] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.826128 339 consensus_queue.cc:260] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:37.827230 32314 consensus_peers.cc:487] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 965f1f0a107b4653b6aac01e0d8e259a. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:37.827445 339 ts_tablet_manager.cc:1428] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:37.827772 339 tablet_bootstrap.cc:492] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.830773 339 tablet_bootstrap.cc:654] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.842005 339 tablet_bootstrap.cc:492] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:37.842255 339 ts_tablet_manager.cc:1397] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20250623 08:04:37.843549 339 raft_consensus.cc:357] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.843798 339 raft_consensus.cc:383] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.843902 339 raft_consensus.cc:738] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.843905 952 ts_tablet_manager.cc:1918] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:37.844035 952 log.cc:1199] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/52d0a356c74f4daea9093c998b4bf73e
I20250623 08:04:37.844350 952 ts_tablet_manager.cc:1939] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:37.844666 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 99) took 5984 ms. Trace:
I20250623 08:04:37.844736 952 rpcz_store.cc:276] 0623 08:04:31.860596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:31.862729 (+ 2133us) service_pool.cc:225] Handling call
0623 08:04:37.734414 (+5871685us) ts_tablet_manager.cc:1145] Deleting tablet 52d0a356c74f4daea9093c998b4bf73e
0623 08:04:37.734424 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.780978 (+ 46554us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.843748 (+ 62770us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.844649 (+ 901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5636749}
I20250623 08:04:37.845041 952 tablet_replica.cc:331] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:37.845211 952 raft_consensus.cc:2241] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:37.845417 952 raft_consensus.cc:2270] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.845613 339 consensus_queue.cc:260] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.846360 339 ts_tablet_manager.cc:1428] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250623 08:04:37.846694 339 tablet_bootstrap.cc:492] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:37.848102 979 ts_tablet_manager.cc:1918] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:37.848289 979 log.cc:1199] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/d6ed4cb252b1452087f8ee53ec78d985
I20250623 08:04:37.848882 979 ts_tablet_manager.cc:1939] T d6ed4cb252b1452087f8ee53ec78d985 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:37.850252 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_63b537810a3a4f478e4226cabee7004d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.851024 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63b537810a3a4f478e4226cabee7004d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.854072 339 tablet_bootstrap.cc:654] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.855517 343 tablet_bootstrap.cc:492] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:37.855638 343 ts_tablet_manager.cc:1397] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20250623 08:04:37.856278 343 raft_consensus.cc:357] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.856436 343 raft_consensus.cc:383] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.856482 343 raft_consensus.cc:738] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.856647 343 consensus_queue.cc:260] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.857055 343 ts_tablet_manager.cc:1428] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250623 08:04:37.857275 343 tablet_bootstrap.cc:492] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:37.858013 32051 master_service.cc:948] ReplaceTablet: received request to replace tablet 358cb6ac24414bdfbc32155e732f9ac8 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:37.858641 343 tablet_bootstrap.cc:654] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.861433 1440 raft_consensus.cc:491] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.861548 1440 raft_consensus.cc:513] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.862049 1440 leader_election.cc:290] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:37.864491 343 tablet_bootstrap.cc:492] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:37.864676 343 ts_tablet_manager.cc:1397] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250623 08:04:37.850175 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 200) took 2746 ms. Trace:
I20250623 08:04:37.845960 952 ts_tablet_manager.cc:1905] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.850488 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 966a159e212040109e2842c15b69fb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.865460 343 raft_consensus.cc:357] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.850800 32498 tablet_service.cc:1515] Processing DeleteTablet for tablet 7da84969a6464b5caeaf47af0cbaaa4d with delete_type TABLET_DATA_DELETED (replaced by tablet 136e482995a24c5daec55a701f3091fa) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.865613 343 raft_consensus.cc:383] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:37.851070 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e47b10f46b8402e94354ebca573c194 with delete_type TABLET_DATA_DELETED (replaced by tablet ea702abb31fb41498ab61bbb05b1f7e2) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.865675 343 raft_consensus.cc:738] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:37.851397 32487 tablet_service.cc:1515] Processing DeleteTablet for tablet 175ee90133994154b7e4b58b0770db3b with delete_type TABLET_DATA_DELETED (replaced by tablet e602d391526d4b0eb0dbf5189afbe075) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.851656 32483 tablet_service.cc:1515] Processing DeleteTablet for tablet 5629d11c44e241d598f3e519b2fe1a44 with delete_type TABLET_DATA_DELETED (replaced by tablet 205da880685b49b3b8669f73dc2c563b) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:37.865844 343 consensus_queue.cc:260] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.851815 32453 leader_election.cc:304] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.866287 343 ts_tablet_manager.cc:1428] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:37.866405 1447 raft_consensus.cc:2802] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:37.866536 343 tablet_bootstrap.cc:492] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:37.866549 1447 raft_consensus.cc:491] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.866637 1447 raft_consensus.cc:3058] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.867226 1575 consensus_queue.cc:1035] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.868077 343 tablet_bootstrap.cc:654] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.869446 1447 raft_consensus.cc:513] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.870698 1447 leader_election.cc:290] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.869596 32453 leader_election.cc:304] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.872362 32453 leader_election.cc:304] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.872402 1447 raft_consensus.cc:2802] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:37.872469 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dc696236a15420e9666cca708068fb2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.872591 1447 raft_consensus.cc:491] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.872625 1575 raft_consensus.cc:2802] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:37.872675 1447 raft_consensus.cc:3058] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.872653 32375 raft_consensus.cc:2466] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:37.872720 1575 raft_consensus.cc:695] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:37.872912 1575 consensus_queue.cc:237] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.873684 32589 leader_election.cc:304] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:37.874354 1440 raft_consensus.cc:2802] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:37.874444 1440 raft_consensus.cc:491] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.874501 1440 raft_consensus.cc:3058] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:37.875595 1447 raft_consensus.cc:513] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.876333 1447 leader_election.cc:290] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.876984 1440 raft_consensus.cc:513] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.876986 32059 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_9c574fd66ea84e8eb1cae99bbb828288" } new_table_name: "default.Table_88ef305b303f49708fc3db029ffb8ace" modify_external_catalogs: true
I20250623 08:04:37.877512 1440 leader_election.cc:290] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:37.877676 1359 consensus_queue.cc:1046] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.878043 1508 raft_consensus.cc:491] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.878142 1508 raft_consensus.cc:513] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.878371 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dc696236a15420e9666cca708068fb2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:37.878542 32375 raft_consensus.cc:3058] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 2
I20250623 08:04:37.878580 1508 leader_election.cc:290] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.878692 1508 raft_consensus.cc:3053] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Stepping down as leader of term 1
I20250623 08:04:37.878741 1508 raft_consensus.cc:738] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:37.878978 1508 consensus_queue.cc:260] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.879206 1508 raft_consensus.cc:3058] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:37.879343 32052 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_5785d39307cb42758ded4062307c55e3" } modify_external_catalogs: true
I20250623 08:04:37.879532 32052 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_5785d39307cb42758ded4062307c55e3" } modify_external_catalogs: true
I20250623 08:04:37.880391 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d496fd69b6f140b88b5abc86c8839ee2" } new_table_name: "default.Table_0b9112de297847f889ab29b68f4dfb8c" modify_external_catalogs: true
I20250623 08:04:37.881361 32375 raft_consensus.cc:2466] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 2.
I20250623 08:04:37.882328 32589 leader_election.cc:304] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:37.883013 1440 raft_consensus.cc:2802] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:37.883219 1440 raft_consensus.cc:695] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 2 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:37.883481 1440 consensus_queue.cc:237] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:37.887017 1582 raft_consensus.cc:491] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.887168 1582 raft_consensus.cc:513] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.887876 1582 leader_election.cc:290] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:37.891121 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.891744 981 ts_tablet_manager.cc:1918] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250623 08:04:37.892349 32453 consensus_peers.cc:487] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:37.892531 32453 leader_election.cc:304] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:37.892683 32589 leader_election.cc:343] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.892728 1359 raft_consensus.cc:2802] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:37.892788 1359 raft_consensus.cc:491] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.892848 1359 raft_consensus.cc:3058] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.895365 1359 raft_consensus.cc:513] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.895893 1359 leader_election.cc:290] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.865070 979 rpcz_store.cc:276] 0623 08:04:35.104012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.104198 (+ 186us) service_pool.cc:225] Handling call
0623 08:04:37.746526 (+2642328us) ts_tablet_manager.cc:1145] Deleting tablet d6ed4cb252b1452087f8ee53ec78d985
0623 08:04:37.746535 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.803329 (+ 56794us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.847940 (+ 44611us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.850152 (+ 2212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2642136}
I20250623 08:04:37.897732 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet d6ed4cb252b1452087f8ee53ec78d985 (table default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]) successfully deleted
I20250623 08:04:37.897820 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 198) took 2794 ms. Trace:
I20250623 08:04:37.897918 979 rpcz_store.cc:276] 0623 08:04:35.103796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.104444 (+ 648us) service_pool.cc:225] Handling call
0623 08:04:37.897693 (+2793249us) ts_tablet_manager.cc:1145] Deleting tablet d4221d7500d1458db75926d7b274a654
0623 08:04:37.897710 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.897800 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2793045}
W20250623 08:04:37.898430 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d4221d7500d1458db75926d7b274a654 because the tablet was not found. No further retry: Not found: Tablet not found: d4221d7500d1458db75926d7b274a654
I20250623 08:04:37.898931 979 tablet_replica.cc:331] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:37.899447 979 raft_consensus.cc:2241] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:37.904068 32703 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:37.924573 32453 leader_election.cc:304] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:37.924822 32453 raft_consensus.cc:2684] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:37.924988 32453 leader_election.cc:304] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:37.925228 1507 raft_consensus.cc:2802] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:37.925307 1507 raft_consensus.cc:491] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.925395 1507 raft_consensus.cc:3058] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.928198 1447 raft_consensus.cc:491] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.928422 1447 raft_consensus.cc:513] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.928880 1507 raft_consensus.cc:513] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.929498 1507 leader_election.cc:290] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.930018 1507 consensus_queue.cc:1035] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:37.930537 1447 leader_election.cc:290] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:37.930912 32453 consensus_peers.cc:487] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f279b63b55c64e548ded27b02025c9f5. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:37.932436 1624 tablet.cc:1722] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:37.938906 979 raft_consensus.cc:2270] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:37.939517 979 ts_tablet_manager.cc:1905] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:37.951613 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:37.951913 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 5 got 0
I20250623 08:04:37.952299 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:37.952746 32058 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
W20250623 08:04:37.954180 32589 leader_election.cc:343] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: Tablet not RUNNING: STOPPED
I20250623 08:04:37.954298 32589 leader_election.cc:304] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.954658 1582 raft_consensus.cc:2747] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:37.955171 32589 leader_election.cc:343] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:37.955267 32589 leader_election.cc:304] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.955598 1582 raft_consensus.cc:2747] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:37.956661 32589 leader_election.cc:304] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.957105 1582 raft_consensus.cc:2802] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:37.957188 1582 raft_consensus.cc:491] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:37.957263 1582 raft_consensus.cc:3058] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.960029 1582 raft_consensus.cc:513] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.965155 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_848fa16d96984425bfb59432e4b51723"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:37.965734 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_848fa16d96984425bfb59432e4b51723 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:37.965890 1624 tablet.cc:1722] T 7adec1482119405daca4aa603770ae22 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:37.968724 32589 leader_election.cc:304] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:37.969090 1506 raft_consensus.cc:2802] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:37.971666 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 52d0a356c74f4daea9093c998b4bf73e (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:37.976823 1507 raft_consensus.cc:491] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.977021 1507 raft_consensus.cc:513] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.977594 1507 leader_election.cc:290] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.985448 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.985702 32648 raft_consensus.cc:2466] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:37.987402 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecb422550efd40cf82bc92fca5c25eda" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:37.987562 32646 raft_consensus.cc:3058] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.987702 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21080ab069d74457a9f7fe3aabdb3c7c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:37.987897 32645 raft_consensus.cc:3058] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.988152 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "724d662bcf754c74a8f3dff91ee258ae" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.988277 32656 raft_consensus.cc:2466] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:37.988785 32652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:37.988950 32652 raft_consensus.cc:2391] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:37.990334 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fd478c007914d2cb09d929b50322a64" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:37.990465 32649 raft_consensus.cc:3058] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:37.990682 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 38b8cfcbce6a472ab2daae28226cf766 (DEFAULT_TABLE table=default.Table_b161fc2726dc4831988f5ac03028b089 [id=c8effd1e07264a5984e51f380a700652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.991077 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b8cfcbce6a472ab2daae28226cf766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.992147 32629 tablet_service.cc:1468] Processing CreateTablet for tablet ee82db25aa544a21a1419e081196f179 (DEFAULT_TABLE table=default.Table_c1548afc4cff490a94e8a03d5fea9339 [id=89148e37f2454772a74b0c88e4164f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:37.992484 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee82db25aa544a21a1419e081196f179. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.998456 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 886fc0fa39c54bddb25ac01f68d95805
I20250623 08:04:37.999058 32057 catalog_manager.cc:6652] ReplaceTablet: tablet 886fc0fa39c54bddb25ac01f68d95805 deleted and replaced by tablet ec4d50d5f9394afe97e3b43d65e6894e
I20250623 08:04:38.011524 32057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_9fc75fd22c6a467782fe3b4c3b123602" } modify_external_catalogs: true
I20250623 08:04:38.011745 32057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_9fc75fd22c6a467782fe3b4c3b123602" } modify_external_catalogs: true
I20250623 08:04:38.012473 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_dd2b39480694444ba170c65257a8dc58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:38.013063 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd2b39480694444ba170c65257a8dc58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:38.013891 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 886fc0fa39c54bddb25ac01f68d95805 with delete_type TABLET_DATA_DELETED (replaced by tablet ec4d50d5f9394afe97e3b43d65e6894e) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:38.014400 1582 leader_election.cc:290] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.019727 32626 tablet_service.cc:1468] Processing CreateTablet for tablet a2a07188a4d340bc9caf010d5202e34d (DEFAULT_TABLE table=default.Table_1ced320c61ec4355bb2686a5c84a55eb [id=61c080721215406397c00aff27538c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.019986 342 tablet_bootstrap.cc:492] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:38.020140 342 ts_tablet_manager.cc:1397] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.204s user 0.001s sys 0.002s
I20250623 08:04:38.020233 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a07188a4d340bc9caf010d5202e34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:37.992595 32643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfd3e263464c4f2c8af9262acf211193" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.021004 342 raft_consensus.cc:357] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.021201 342 raft_consensus.cc:383] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.021258 342 raft_consensus.cc:738] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.021446 342 consensus_queue.cc:260] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:37.909905 1332 raft_consensus.cc:491] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:37.923682 339 tablet_bootstrap.cc:492] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.021904 339 ts_tablet_manager.cc:1397] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.175s user 0.006s sys 0.000s
I20250623 08:04:38.021857 1332 raft_consensus.cc:513] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.022523 1332 leader_election.cc:290] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.022647 339 raft_consensus.cc:357] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.022826 339 raft_consensus.cc:383] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.022879 339 raft_consensus.cc:738] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.023041 339 consensus_queue.cc:260] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.023514 339 ts_tablet_manager.cc:1428] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:38.023772 339 tablet_bootstrap.cc:492] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.025311 339 tablet_bootstrap.cc:654] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:37.937070 1622 raft_consensus.cc:491] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.026134 1622 raft_consensus.cc:513] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.026276 32646 raft_consensus.cc:2466] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:38.026727 1622 leader_election.cc:290] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.952912 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:38.026930 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "205da880685b49b3b8669f73dc2c563b" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:37.954306 1303 raft_consensus.cc:491] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.027339 1303 raft_consensus.cc:513] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:38.027434 32314 leader_election.cc:343] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: Tablet not RUNNING: STOPPED
I20250623 08:04:38.027843 1303 leader_election.cc:290] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:37.971625 32376 raft_consensus.cc:3058] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.030431 981 log.cc:1199] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/ea702abb31fb41498ab61bbb05b1f7e2
I20250623 08:04:38.030612 32643 raft_consensus.cc:2466] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:38.030740 981 ts_tablet_manager.cc:1939] T ea702abb31fb41498ab61bbb05b1f7e2 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:37.972543 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecb422550efd40cf82bc92fca5c25eda" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:37.972930 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fd478c007914d2cb09d929b50322a64" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:38.031517 32314 leader_election.cc:343] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 1a070895f4984493a3b9f6fa779797a9
I20250623 08:04:37.973433 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "724d662bcf754c74a8f3dff91ee258ae" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.973906 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 38b8cfcbce6a472ab2daae28226cf766 (DEFAULT_TABLE table=default.Table_b161fc2726dc4831988f5ac03028b089 [id=c8effd1e07264a5984e51f380a700652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.032126 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b8cfcbce6a472ab2daae28226cf766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.034134 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1a070895f4984493a3b9f6fa779797a9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:37.974254 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:37.974999 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21080ab069d74457a9f7fe3aabdb3c7c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:37.975421 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfd3e263464c4f2c8af9262acf211193" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:37.975987 32350 tablet_service.cc:1468] Processing CreateTablet for tablet ee82db25aa544a21a1419e081196f179 (DEFAULT_TABLE table=default.Table_c1548afc4cff490a94e8a03d5fea9339 [id=89148e37f2454772a74b0c88e4164f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.035187 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee82db25aa544a21a1419e081196f179. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.037096 32428 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:37.977792 32348 tablet_service.cc:1468] Processing CreateTablet for tablet a2a07188a4d340bc9caf010d5202e34d (DEFAULT_TABLE table=default.Table_1ced320c61ec4355bb2686a5c84a55eb [id=61c080721215406397c00aff27538c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.039053 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a07188a4d340bc9caf010d5202e34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.040182 952 ts_tablet_manager.cc:1918] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250623 08:04:38.039750 32453 leader_election.cc:343] T 52d0a356c74f4daea9093c998b4bf73e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.041013 952 log.cc:1199] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a9c8d480b8ba49a1be140b0e0a79f4e4
I20250623 08:04:38.005198 32346 tablet_service.cc:1515] Processing DeleteTablet for tablet 886fc0fa39c54bddb25ac01f68d95805 with delete_type TABLET_DATA_DELETED (replaced by tablet ec4d50d5f9394afe97e3b43d65e6894e) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.043776 952 ts_tablet_manager.cc:1939] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:38.046970 32314 leader_election.cc:304] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.047281 1332 raft_consensus.cc:2747] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:38.047698 32314 leader_election.cc:343] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 162308da898d4dde8ab5b1ab01d0f393
W20250623 08:04:38.048002 32314 leader_election.cc:343] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 205da880685b49b3b8669f73dc2c563b
I20250623 08:04:38.048144 32314 leader_election.cc:304] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.048360 1332 raft_consensus.cc:2747] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:38.048745 32314 leader_election.cc:343] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 162308da898d4dde8ab5b1ab01d0f393
I20250623 08:04:38.048887 32314 leader_election.cc:304] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.048463 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 100) took 5626 ms. Trace:
I20250623 08:04:38.049103 1332 raft_consensus.cc:2747] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.049774 952 rpcz_store.cc:276] 0623 08:04:32.422021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.422069 (+ 48us) service_pool.cc:225] Handling call
0623 08:04:37.844995 (+5422926us) ts_tablet_manager.cc:1145] Deleting tablet a9c8d480b8ba49a1be140b0e0a79f4e4
0623 08:04:37.845005 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.964354 (+119349us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.039295 (+ 74941us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.048443 (+ 9148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5422718}
I20250623 08:04:38.052651 339 tablet_bootstrap.cc:492] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.052778 339 ts_tablet_manager.cc:1397] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250623 08:04:38.053521 339 raft_consensus.cc:357] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.053673 339 raft_consensus.cc:383] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.053725 339 raft_consensus.cc:738] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.053933 339 consensus_queue.cc:260] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.054270 952 tablet_replica.cc:331] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:38.054394 339 ts_tablet_manager.cc:1428] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:38.054636 339 tablet_bootstrap.cc:492] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.055888 952 raft_consensus.cc:2241] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.056421 339 tablet_bootstrap.cc:654] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.057807 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 164) took 3397 ms. Trace:
I20250623 08:04:38.057890 981 rpcz_store.cc:276] 0623 08:04:34.660490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.660536 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:37.626868 (+2966332us) ts_tablet_manager.cc:1145] Deleting tablet ea702abb31fb41498ab61bbb05b1f7e2
0623 08:04:37.626878 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.821289 (+194411us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:37.891617 (+ 70328us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.057792 (+166175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2966117}
I20250623 08:04:38.058342 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet ea702abb31fb41498ab61bbb05b1f7e2 (table default.Table_8b1e8627d7a64cc394552d28e372c826 [id=ad04e798828e40359cbe83876ebf505e]) successfully deleted
I20250623 08:04:38.060627 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:38.065035 32350 tablet_service.cc:1468] Processing CreateTablet for tablet ba5913811e9343389c9ea0ec8b8c05ef (DEFAULT_TABLE table=default.Table_787415ae5937475f90f90000aa575d0e [id=4b27489168854629848531915104847a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.065500 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba5913811e9343389c9ea0ec8b8c05ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.070626 32645 raft_consensus.cc:2466] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:38.083550 32649 raft_consensus.cc:2466] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:38.085271 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_de7a3df7f0064a7bb7715c0c5dcc337e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:38.086071 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de7a3df7f0064a7bb7715c0c5dcc337e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:38.086946 339 tablet_bootstrap.cc:492] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.087141 339 ts_tablet_manager.cc:1397] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.003s
I20250623 08:04:38.087961 339 raft_consensus.cc:357] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.088199 339 raft_consensus.cc:383] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.088339 339 raft_consensus.cc:738] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.088620 339 consensus_queue.cc:260] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.090255 339 ts_tablet_manager.cc:1428] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:38.092569 343 tablet_bootstrap.cc:492] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.092706 343 ts_tablet_manager.cc:1397] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.226s user 0.027s sys 0.030s
I20250623 08:04:38.093542 343 raft_consensus.cc:357] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.093709 343 raft_consensus.cc:383] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.093765 343 raft_consensus.cc:738] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.093973 343 consensus_queue.cc:260] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.094409 343 ts_tablet_manager.cc:1428] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:38.094657 343 tablet_bootstrap.cc:492] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:38.095337 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_ddd07329c08c4b6e8468335182c7c7e9" } new_table_name: "default.Table_f8d48385e5b1441bb296f68cee25e5b2" modify_external_catalogs: true
I20250623 08:04:38.095549 979 ts_tablet_manager.cc:1918] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:38.095674 979 log.cc:1199] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/205da880685b49b3b8669f73dc2c563b
I20250623 08:04:38.095986 979 ts_tablet_manager.cc:1939] T 205da880685b49b3b8669f73dc2c563b P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.096711 343 tablet_bootstrap.cc:654] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:38.098099 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 128457us
I20250623 08:04:38.098338 1633 tablet.cc:1722] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:38.099182 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:38.098647 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 197) took 2994 ms. Trace:
I20250623 08:04:38.063014 981 tablet_replica.cc:331] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:38.099871 981 raft_consensus.cc:2241] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.100095 981 raft_consensus.cc:2270] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.099455 979 rpcz_store.cc:276] 0623 08:04:35.103718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.104670 (+ 952us) service_pool.cc:225] Handling call
0623 08:04:37.898809 (+2794139us) ts_tablet_manager.cc:1145] Deleting tablet 205da880685b49b3b8669f73dc2c563b
0623 08:04:37.898851 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:37.942683 (+ 43832us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.095432 (+152749us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.098631 (+ 3199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2793953}
I20250623 08:04:38.100852 1582 raft_consensus.cc:491] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.100961 1582 raft_consensus.cc:513] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.101462 1582 leader_election.cc:290] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.102706 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 196) took 2998 ms. Trace:
I20250623 08:04:38.104312 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f8aa424bbb4aa8a907259270d46656" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:38.104372 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 205da880685b49b3b8669f73dc2c563b (table default.Table_4d4535cab44545839eb3a559385e395a [id=bb849e86d7084959830c68207fb99061]) successfully deleted
I20250623 08:04:38.103596 979 rpcz_store.cc:276] 0623 08:04:35.103629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.104873 (+ 1244us) service_pool.cc:225] Handling call
0623 08:04:38.101860 (+2996987us) ts_tablet_manager.cc:1145] Deleting tablet cde7df7d39a54114a4dcb28cfd1eb429
0623 08:04:38.101869 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.102575 (+ 706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2996714}
I20250623 08:04:38.106889 1506 raft_consensus.cc:695] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:38.107182 1506 consensus_queue.cc:237] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:38.106946 32589 leader_election.cc:343] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:38.107641 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet cde7df7d39a54114a4dcb28cfd1eb429 because the tablet was not found. No further retry: Not found: Tablet not found: cde7df7d39a54114a4dcb28cfd1eb429
I20250623 08:04:38.107594 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 199) took 3003 ms. Trace:
W20250623 08:04:38.108716 32589 leader_election.cc:343] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.108403 979 rpcz_store.cc:276] 0623 08:04:35.103910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.106186 (+ 2276us) service_pool.cc:225] Handling call
0623 08:04:38.106682 (+3000496us) ts_tablet_manager.cc:1145] Deleting tablet 696f5d2797ef463bb72a2119501c6ef3
0623 08:04:38.106694 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.107223 (+ 529us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3000209}
I20250623 08:04:38.110579 342 ts_tablet_manager.cc:1428] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.090s user 0.000s sys 0.002s
I20250623 08:04:38.110934 342 tablet_bootstrap.cc:492] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:38.115386 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 75996us
W20250623 08:04:38.115806 32453 leader_election.cc:343] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.115900 1507 consensus_queue.cc:1035] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:38.120024 32453 consensus_peers.cc:487] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ffafd10f26740c9a5b6c9bdc3bb23a0. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:38.122443 32453 leader_election.cc:343] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:38.123065 32453 leader_election.cc:343] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.123351 32348 tablet_service.cc:1468] Processing CreateTablet for tablet c2c990297f554adbb2d97f6309758940 (DEFAULT_TABLE table=default.Table_3cbae8872bcd4af3a1a837940945fab6 [id=0ea2aee143054b39ae96e30c64a42a50]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:38.123660 32453 leader_election.cc:343] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:38.124140 32453 leader_election.cc:343] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.125001 32355 tablet_service.cc:1468] Processing CreateTablet for tablet b4e01b430a064740b76e0bceec4f5546 (DEFAULT_TABLE table=default.Table_bcdbc5ed766f43ba91aaa4711f0b24f6 [id=7a2cd6d0739b48dca9ef91af18569181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.125799 1575 consensus_queue.cc:1035] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:38.126571 32453 leader_election.cc:343] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.126646 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 1ae0a411a3e54c45b8d5c5b9b0eea762 (DEFAULT_TABLE table=default.Table_71f2f0414e0840be832096955576cb2f [id=a6918c5eb39a435ab4b035c1509916a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.128540 1332 raft_consensus.cc:491] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.128682 1332 raft_consensus.cc:513] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.129166 1332 leader_election.cc:290] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:38.127152 32453 leader_election.cc:343] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:38.129612 32453 leader_election.cc:343] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.131132 981 ts_tablet_manager.cc:1905] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:38.140172 32314 leader_election.cc:343] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 75a6ebe0d6b0463b8ddcb1a091a2cc6f
W20250623 08:04:38.140508 32314 leader_election.cc:343] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 75a6ebe0d6b0463b8ddcb1a091a2cc6f
I20250623 08:04:38.141176 32314 leader_election.cc:304] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.141459 1332 raft_consensus.cc:2747] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.142189 952 raft_consensus.cc:2270] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.142490 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.142665 32514 raft_consensus.cc:2466] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:38.142830 952 ts_tablet_manager.cc:1905] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.143316 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:38.144762 32514 raft_consensus.cc:2391] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:38.143659 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f8aa424bbb4aa8a907259270d46656" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.147260 339 tablet_bootstrap.cc:492] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.147435 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2c990297f554adbb2d97f6309758940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.149057 339 tablet_bootstrap.cc:654] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.151103 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e01b430a064740b76e0bceec4f5546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.155010 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ae0a411a3e54c45b8d5c5b9b0eea762. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.155629 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:38.111119 343 tablet_bootstrap.cc:492] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.127943 32630 tablet_service.cc:1468] Processing CreateTablet for tablet c2c990297f554adbb2d97f6309758940 (DEFAULT_TABLE table=default.Table_3cbae8872bcd4af3a1a837940945fab6 [id=0ea2aee143054b39ae96e30c64a42a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.162314 343 ts_tablet_manager.cc:1397] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.001s
I20250623 08:04:38.162533 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2c990297f554adbb2d97f6309758940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.163185 343 raft_consensus.cc:357] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.163347 343 raft_consensus.cc:383] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.163395 343 raft_consensus.cc:738] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.163559 343 consensus_queue.cc:260] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.163988 343 ts_tablet_manager.cc:1428] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:38.164266 343 tablet_bootstrap.cc:492] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:38.165848 343 tablet_bootstrap.cc:654] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.128026 32626 tablet_service.cc:1468] Processing CreateTablet for tablet ba5913811e9343389c9ea0ec8b8c05ef (DEFAULT_TABLE table=default.Table_787415ae5937475f90f90000aa575d0e [id=4b27489168854629848531915104847a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.167100 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba5913811e9343389c9ea0ec8b8c05ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.167145 952 ts_tablet_manager.cc:1918] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:38.167270 952 log.cc:1199] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/e9f8aa424bbb4aa8a907259270d46656
I20250623 08:04:38.167565 952 ts_tablet_manager.cc:1939] T e9f8aa424bbb4aa8a907259270d46656 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:38.167860 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 102) took 5744 ms. Trace:
I20250623 08:04:38.167945 952 rpcz_store.cc:276] 0623 08:04:32.422976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.423016 (+ 40us) service_pool.cc:225] Handling call
0623 08:04:38.054197 (+5631181us) ts_tablet_manager.cc:1145] Deleting tablet e9f8aa424bbb4aa8a907259270d46656
0623 08:04:38.054210 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.146971 (+ 92761us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.167039 (+ 20068us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.167844 (+ 805us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85423,"tablet-delete.queue_time_us":5630961}
I20250623 08:04:38.168291 952 tablet_replica.cc:331] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:38.168488 952 raft_consensus.cc:2241] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.168660 952 raft_consensus.cc:2270] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.169278 952 ts_tablet_manager.cc:1905] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.128314 32629 tablet_service.cc:1468] Processing CreateTablet for tablet b4e01b430a064740b76e0bceec4f5546 (DEFAULT_TABLE table=default.Table_bcdbc5ed766f43ba91aaa4711f0b24f6 [id=7a2cd6d0739b48dca9ef91af18569181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.150027 1633 tablet.cc:1722] T 7adec1482119405daca4aa603770ae22 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:38.171144 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e01b430a064740b76e0bceec4f5546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.171864 339 tablet_bootstrap.cc:492] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.171980 339 ts_tablet_manager.cc:1397] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.006s
I20250623 08:04:38.173612 339 raft_consensus.cc:357] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.173766 339 raft_consensus.cc:383] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.173817 339 raft_consensus.cc:738] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.174005 339 consensus_queue.cc:260] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:38.174129 32054 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c529e826fa84a919c018887620e4023 has no consensus state
I20250623 08:04:38.150836 32650 raft_consensus.cc:1273] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Refusing update from remote peer 59f7c614254f42009306b13337f06576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 08:04:38.151533 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.174515 32642 raft_consensus.cc:2466] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:38.175079 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_62ddca96ee344b168687745f9ef92af4" } modify_external_catalogs: true
I20250623 08:04:38.175351 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_62ddca96ee344b168687745f9ef92af4" } modify_external_catalogs: true
I20250623 08:04:38.176452 339 ts_tablet_manager.cc:1428] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:38.176774 339 tablet_bootstrap.cc:492] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.178576 339 tablet_bootstrap.cc:654] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.183933 979 tablet_replica.cc:331] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.184120 979 raft_consensus.cc:2241] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:38.184542 32054 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 47f83b55012f46f7abda1f2e28104c1e
I20250623 08:04:38.185343 32054 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_094f2e2ebe104774b6a713a5bbb7fd2f" } modify_external_catalogs: true
I20250623 08:04:38.185614 32054 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_094f2e2ebe104774b6a713a5bbb7fd2f" } modify_external_catalogs: true
W20250623 08:04:38.186497 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 696f5d2797ef463bb72a2119501c6ef3 because the tablet was not found. No further retry: Not found: Tablet not found: 696f5d2797ef463bb72a2119501c6ef3
I20250623 08:04:38.192616 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet a9c8d480b8ba49a1be140b0e0a79f4e4 (table default.Table_4550e74cc6e845719c1466d3acd1b723 [id=587d6896c7dd40838ce8ad01026379a8]) successfully deleted
I20250623 08:04:38.193426 32374 raft_consensus.cc:3058] T 0c58231f67604422a1497e4c1b44ec52 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:38.197499 32589 leader_election.cc:343] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 8dc696236a15420e9666cca708068fb2
W20250623 08:04:38.197837 32589 leader_election.cc:343] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 8dc696236a15420e9666cca708068fb2
I20250623 08:04:38.198181 32589 leader_election.cc:304] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.201623 1634 raft_consensus.cc:2802] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:38.201706 1634 raft_consensus.cc:491] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:38.201761 1634 raft_consensus.cc:3058] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.205183 1634 raft_consensus.cc:513] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.205933 1634 leader_election.cc:290] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.206703 979 pending_rounds.cc:70] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:38.206826 979 pending_rounds.cc:77] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7170727435813101568 op_type: NO_OP noop_request { }
I20250623 08:04:38.206990 979 raft_consensus.cc:2887] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:38.207020 342 tablet_bootstrap.cc:654] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.207338 1332 raft_consensus.cc:491] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.207500 1332 raft_consensus.cc:513] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.208202 1332 leader_election.cc:290] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.207157 979 raft_consensus.cc:2270] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.209303 979 ts_tablet_manager.cc:1905] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:38.209705 32314 leader_election.cc:343] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 6c6f98e2db1a4b0789c5819b2c7d4bb6
I20250623 08:04:38.210813 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 1ae0a411a3e54c45b8d5c5b9b0eea762 (DEFAULT_TABLE table=default.Table_71f2f0414e0840be832096955576cb2f [id=a6918c5eb39a435ab4b035c1509916a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.211211 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ae0a411a3e54c45b8d5c5b9b0eea762. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.215725 1643 tablet.cc:1722] T 7adec1482119405daca4aa603770ae22 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:38.216661 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 7f416cdb596a4ed5a302acd07bb84b9c (DEFAULT_TABLE table=default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.217083 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f416cdb596a4ed5a302acd07bb84b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.217708 979 ts_tablet_manager.cc:1918] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250623 08:04:38.217818 979 log.cc:1199] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/8dc696236a15420e9666cca708068fb2
I20250623 08:04:38.218163 979 ts_tablet_manager.cc:1939] T 8dc696236a15420e9666cca708068fb2 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.218477 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 203) took 2940 ms. Trace:
I20250623 08:04:38.218549 979 rpcz_store.cc:276] 0623 08:04:35.278326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.278641 (+ 315us) service_pool.cc:225] Handling call
0623 08:04:38.183885 (+2905244us) ts_tablet_manager.cc:1145] Deleting tablet 8dc696236a15420e9666cca708068fb2
0623 08:04:38.183893 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.212303 (+ 28410us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.217619 (+ 5316us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.218463 (+ 844us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22306,"tablet-delete.queue_time_us":2905031}
I20250623 08:04:38.218966 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 204) took 2940 ms. Trace:
I20250623 08:04:38.219045 979 rpcz_store.cc:276] 0623 08:04:35.278420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.278890 (+ 470us) service_pool.cc:225] Handling call
0623 08:04:38.218878 (+2939988us) ts_tablet_manager.cc:1145] Deleting tablet f3d32798168345edb9898751d083057c
0623 08:04:38.218888 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.218951 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2939840}
I20250623 08:04:38.219352 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 205) took 2940 ms. Trace:
I20250623 08:04:38.219424 979 rpcz_store.cc:276] 0623 08:04:35.278567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.279073 (+ 506us) service_pool.cc:225] Handling call
0623 08:04:38.219278 (+2940205us) ts_tablet_manager.cc:1145] Deleting tablet d0ee5a4fe03848229524cc0105226d8c
0623 08:04:38.219288 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.219339 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2940022}
I20250623 08:04:38.219666 979 tablet_replica.cc:331] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.219847 979 raft_consensus.cc:2241] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.220012 979 raft_consensus.cc:2270] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.220527 979 ts_tablet_manager.cc:1905] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.225246 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 8f8f8a132e6245f9a04b5bb6f2253c76 (DEFAULT_TABLE table=default.Table_c748f600d56f4374bdb31ad502ec7557 [id=3b342fcf5bff4bf0bb19c90fca9015c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.225567 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8f8a132e6245f9a04b5bb6f2253c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.229095 342 tablet_bootstrap.cc:492] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:38.229247 342 ts_tablet_manager.cc:1397] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.000s
I20250623 08:04:38.231822 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb99da3b8fa4dc6aa7f11a829d3f2d2 (DEFAULT_TABLE table=default.Table_3980cd736bd342f4bcd5cf289c1c8c25 [id=47a9c2f4140743c3a357032721181f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.232134 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb99da3b8fa4dc6aa7f11a829d3f2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.238683 981 ts_tablet_manager.cc:1918] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:38.238905 981 log.cc:1199] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/2ecf4af9e476495683bd63cac20d4672
I20250623 08:04:38.239362 981 ts_tablet_manager.cc:1939] T 2ecf4af9e476495683bd63cac20d4672 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:38.239774 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 165) took 3578 ms. Trace:
I20250623 08:04:38.239931 981 rpcz_store.cc:276] 0623 08:04:34.660932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.660973 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:38.062958 (+3401985us) ts_tablet_manager.cc:1145] Deleting tablet 2ecf4af9e476495683bd63cac20d4672
0623 08:04:38.062970 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.135343 (+ 72373us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.238532 (+103189us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.239754 (+ 1222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3401836}
I20250623 08:04:38.240382 981 tablet_replica.cc:331] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:38.240578 32629 tablet_service.cc:1468] Processing CreateTablet for tablet e41b15c89a744a698ca5eee2f143b4bc (DEFAULT_TABLE table=default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.240686 981 raft_consensus.cc:2241] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.240880 981 raft_consensus.cc:2270] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.241108 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41b15c89a744a698ca5eee2f143b4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.241539 981 ts_tablet_manager.cc:1905] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.243757 343 tablet_bootstrap.cc:492] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.243932 343 ts_tablet_manager.cc:1397] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.015s
I20250623 08:04:38.244771 343 raft_consensus.cc:357] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.245021 343 raft_consensus.cc:383] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.245116 343 raft_consensus.cc:738] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.245388 343 consensus_queue.cc:260] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.245934 343 ts_tablet_manager.cc:1428] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:38.246250 343 tablet_bootstrap.cc:492] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:38.246888 1332 raft_consensus.cc:491] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.247037 1332 raft_consensus.cc:513] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.247638 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 3bfad68d1f20486997ea1b0a8d97206b (DEFAULT_TABLE table=default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.247747 1332 leader_election.cc:290] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.247948 343 tablet_bootstrap.cc:654] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.249493 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 8dc696236a15420e9666cca708068fb2 (table default.Table_f81a1165815543b781251c1ecfd06779 [id=491b95490f8c45eda181ea0cafd42368]) successfully deleted
W20250623 08:04:38.249817 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f3d32798168345edb9898751d083057c because the tablet was not found. No further retry: Not found: Tablet not found: f3d32798168345edb9898751d083057c
I20250623 08:04:38.247967 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bfad68d1f20486997ea1b0a8d97206b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.252375 339 tablet_bootstrap.cc:492] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.252513 339 ts_tablet_manager.cc:1397] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.000s
I20250623 08:04:38.248559 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 1494f83d13cd49c7a804593f45f97cc6 (DEFAULT_TABLE table=default.Table_d87c4052e62d4c1c87fc25cff314319b [id=58d5dec0c723470a9a7e56ce37dc71d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.252976 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1494f83d13cd49c7a804593f45f97cc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.255957 1649 tablet.cc:1722] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:38.265190 339 raft_consensus.cc:357] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.266808 339 raft_consensus.cc:383] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.266942 339 raft_consensus.cc:738] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.267197 339 consensus_queue.cc:260] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.248852 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 1cee3de5953244f98428747bea912f77 (DEFAULT_TABLE table=default.Table_969668212d4f422ca8f2f34266315161 [id=eb6d9c5745e649978f87c97627ace9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.250533 979 ts_tablet_manager.cc:1918] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.267732 979 log.cc:1199] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/f4a0b9402a264800a1ea39707c10da0c
I20250623 08:04:38.267793 339 ts_tablet_manager.cc:1428] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250623 08:04:38.268049 339 tablet_bootstrap.cc:492] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.268082 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cee3de5953244f98428747bea912f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.268136 979 ts_tablet_manager.cc:1939] T f4a0b9402a264800a1ea39707c10da0c P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.268509 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 202) took 2990 ms. Trace:
I20250623 08:04:38.268592 979 rpcz_store.cc:276] 0623 08:04:35.278178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.279288 (+ 1110us) service_pool.cc:225] Handling call
0623 08:04:38.219615 (+2940327us) ts_tablet_manager.cc:1145] Deleting tablet f4a0b9402a264800a1ea39707c10da0c
0623 08:04:38.219625 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.238315 (+ 18690us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.250443 (+ 12128us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.268495 (+ 18052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2940181}
I20250623 08:04:38.269095 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 206) took 2760 ms. Trace:
I20250623 08:04:38.269178 979 rpcz_store.cc:276] 0623 08:04:35.509057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.509111 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:38.268896 (+2759785us) ts_tablet_manager.cc:1145] Deleting tablet 501b5233522f43c7aed3c9770adb5c03
0623 08:04:38.268906 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.269081 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2715793}
I20250623 08:04:38.269507 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 207) took 2760 ms. Trace:
I20250623 08:04:38.269587 979 rpcz_store.cc:276] 0623 08:04:35.509372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.509415 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:38.269409 (+2759994us) ts_tablet_manager.cc:1145] Deleting tablet 789d1c20aa72480ab71123c799bbde4c
0623 08:04:38.269420 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.269494 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2716249}
I20250623 08:04:38.269887 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 208) took 2747 ms. Trace:
I20250623 08:04:38.269958 979 rpcz_store.cc:276] 0623 08:04:35.522009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.522062 (+ 53us) service_pool.cc:225] Handling call
0623 08:04:38.269790 (+2747728us) ts_tablet_manager.cc:1145] Deleting tablet db81250d1e164a7d94affd764fc0ff0c
0623 08:04:38.269800 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.269875 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2716586}
I20250623 08:04:38.270263 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 212) took 2278 ms. Trace:
I20250623 08:04:38.270335 979 rpcz_store.cc:276] 0623 08:04:35.991995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.993567 (+ 1572us) service_pool.cc:225] Handling call
0623 08:04:38.270164 (+2276597us) ts_tablet_manager.cc:1145] Deleting tablet d19d468ce13d4108a9badc36fc503306
0623 08:04:38.270174 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.270251 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2276389}
I20250623 08:04:38.270568 979 tablet_replica.cc:331] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.270802 979 raft_consensus.cc:2241] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.270969 979 raft_consensus.cc:2270] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.271450 979 ts_tablet_manager.cc:1905] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.270740 339 tablet_bootstrap.cc:654] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.275943 952 ts_tablet_manager.cc:1918] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:38.276435 952 log.cc:1199] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/ecb422550efd40cf82bc92fca5c25eda
I20250623 08:04:38.277033 952 ts_tablet_manager.cc:1939] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:38.277127 979 ts_tablet_manager.cc:1918] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:38.277238 979 log.cc:1199] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/18ea63263b9b466e922169eecafab747
I20250623 08:04:38.277519 979 ts_tablet_manager.cc:1939] T 18ea63263b9b466e922169eecafab747 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.277666 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 103) took 5854 ms. Trace:
I20250623 08:04:38.277796 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 213) took 2281 ms. Trace:
I20250623 08:04:38.277894 979 rpcz_store.cc:276] 0623 08:04:35.996204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.996262 (+ 58us) service_pool.cc:225] Handling call
0623 08:04:38.270527 (+2274265us) ts_tablet_manager.cc:1145] Deleting tablet 18ea63263b9b466e922169eecafab747
0623 08:04:38.270537 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.274379 (+ 3842us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.277056 (+ 2677us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.277778 (+ 722us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2274068}
I20250623 08:04:38.277815 952 rpcz_store.cc:276] 0623 08:04:32.423418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.423463 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:38.168248 (+5744785us) ts_tablet_manager.cc:1145] Deleting tablet ecb422550efd40cf82bc92fca5c25eda
0623 08:04:38.168256 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.186147 (+ 17891us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.275819 (+ 89672us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.277629 (+ 1810us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5744608}
I20250623 08:04:38.278218 979 tablet_replica.cc:331] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.278389 979 raft_consensus.cc:2241] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.278527 952 tablet_replica.cc:331] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:38.278551 979 raft_consensus.cc:2270] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.279611 339 tablet_bootstrap.cc:492] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.279800 339 ts_tablet_manager.cc:1397] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20250623 08:04:38.280558 339 raft_consensus.cc:357] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.280922 339 raft_consensus.cc:383] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.281023 339 raft_consensus.cc:738] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.281265 339 consensus_queue.cc:260] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.281886 339 ts_tablet_manager.cc:1428] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:38.282192 339 tablet_bootstrap.cc:492] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:38.283640 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 85474.6us
I20250623 08:04:38.283768 979 ts_tablet_manager.cc:1905] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.284047 339 tablet_bootstrap.cc:654] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.284238 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:38.285004 32589 leader_election.cc:343] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.270490 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
W20250623 08:04:38.286209 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d0ee5a4fe03848229524cc0105226d8c because the tablet was not found. No further retry: Not found: Tablet not found: d0ee5a4fe03848229524cc0105226d8c
I20250623 08:04:38.286608 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:38.288358 32342 tablet_service.cc:1468] Processing CreateTablet for tablet 1cee3de5953244f98428747bea912f77 (DEFAULT_TABLE table=default.Table_969668212d4f422ca8f2f34266315161 [id=eb6d9c5745e649978f87c97627ace9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.288586 32348 tablet_service.cc:1468] Processing CreateTablet for tablet e41b15c89a744a698ca5eee2f143b4bc (DEFAULT_TABLE table=default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.288825 32342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cee3de5953244f98428747bea912f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.289048 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41b15c89a744a698ca5eee2f143b4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.288353 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb99da3b8fa4dc6aa7f11a829d3f2d2 (DEFAULT_TABLE table=default.Table_3980cd736bd342f4bcd5cf289c1c8c25 [id=47a9c2f4140743c3a357032721181f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.291178 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 7f416cdb596a4ed5a302acd07bb84b9c (DEFAULT_TABLE table=default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.291359 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb99da3b8fa4dc6aa7f11a829d3f2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.291608 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f416cdb596a4ed5a302acd07bb84b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.293146 339 tablet_bootstrap.cc:492] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.293316 339 ts_tablet_manager.cc:1397] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250623 08:04:38.294253 339 raft_consensus.cc:357] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.294479 339 raft_consensus.cc:383] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.294579 339 raft_consensus.cc:738] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.294878 339 consensus_queue.cc:260] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.294991 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 8f8f8a132e6245f9a04b5bb6f2253c76 (DEFAULT_TABLE table=default.Table_c748f600d56f4374bdb31ad502ec7557 [id=3b342fcf5bff4bf0bb19c90fca9015c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.295310 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8f8a132e6245f9a04b5bb6f2253c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.295639 339 ts_tablet_manager.cc:1428] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:38.296061 339 tablet_bootstrap.cc:492] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.297639 339 tablet_bootstrap.cc:654] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.305495 339 tablet_bootstrap.cc:492] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.305721 339 ts_tablet_manager.cc:1397] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20250623 08:04:38.306577 339 raft_consensus.cc:357] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.306838 339 raft_consensus.cc:383] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.306928 339 raft_consensus.cc:738] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.307163 339 consensus_queue.cc:260] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.307754 339 ts_tablet_manager.cc:1428] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:38.308074 339 tablet_bootstrap.cc:492] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.309716 339 tablet_bootstrap.cc:654] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.312098 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:38.312578 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_bc1251ce708a4467b5b1c71a1df307e0 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 1 got 0
I20250623 08:04:38.312839 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:38.313088 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:38.313694 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:38.313864 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:38.314495 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 90ceb0f8d5c94c4e9427042d8a35e16c (table default.Table_0972bb6b77644e27b56f3e0007bc5fd4 [id=6fa96454486845848e54e86fd64a0017]). Expected version 1 got 0
I20250623 08:04:38.287017 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet f4a0b9402a264800a1ea39707c10da0c (table default.Table_244bcdc8d69b4f61a337dac403ffe6cb [id=b3ab55b66d754642aed2f746d9a35807]) successfully deleted
W20250623 08:04:38.315146 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 501b5233522f43c7aed3c9770adb5c03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 501b5233522f43c7aed3c9770adb5c03 already in progress: creating tablet
W20250623 08:04:38.315418 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 789d1c20aa72480ab71123c799bbde4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 789d1c20aa72480ab71123c799bbde4c already in progress: creating tablet
W20250623 08:04:38.315747 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet db81250d1e164a7d94affd764fc0ff0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db81250d1e164a7d94affd764fc0ff0c already in progress: creating tablet
W20250623 08:04:38.315977 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d19d468ce13d4108a9badc36fc503306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d19d468ce13d4108a9badc36fc503306 already in progress: creating tablet
I20250623 08:04:38.316200 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 18ea63263b9b466e922169eecafab747 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]) successfully deleted
I20250623 08:04:38.316860 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 2ecf4af9e476495683bd63cac20d4672 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:38.319792 339 tablet_bootstrap.cc:492] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.319990 339 ts_tablet_manager.cc:1397] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20250623 08:04:38.320781 339 raft_consensus.cc:357] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.320997 339 raft_consensus.cc:383] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.321276 339 raft_consensus.cc:738] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.321519 339 consensus_queue.cc:260] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.322134 339 ts_tablet_manager.cc:1428] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:38.322592 339 tablet_bootstrap.cc:492] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.324225 339 tablet_bootstrap.cc:654] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.311451 342 raft_consensus.cc:357] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.324898 1508 raft_consensus.cc:491] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.324973 342 raft_consensus.cc:383] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.325047 342 raft_consensus.cc:738] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.325076 1508 raft_consensus.cc:513] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.325245 342 consensus_queue.cc:260] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.325825 1508 leader_election.cc:290] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.326025 342 ts_tablet_manager.cc:1428] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.097s user 0.001s sys 0.001s
I20250623 08:04:38.311736 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c6f98e2db1a4b0789c5819b2c7d4bb6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.311712 1627 raft_consensus.cc:491] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.326574 1627 raft_consensus.cc:513] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.326722 952 raft_consensus.cc:2241] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.327422 1627 leader_election.cc:290] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.326457 342 tablet_bootstrap.cc:492] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:38.330117 342 tablet_bootstrap.cc:654] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.330580 981 ts_tablet_manager.cc:1918] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.331106 952 raft_consensus.cc:2270] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.331750 32453 leader_election.cc:304] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.332456 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 59015204545e436cb558fe4465a67c97 (table default.Table_094f2e2ebe104774b6a713a5bbb7fd2f [id=9979c6d8878443c09e6ff9f4aea65d62]). Expected version 1 got 0
I20250623 08:04:38.333755 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:38.334057 1359 raft_consensus.cc:2802] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:38.334982 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:38.336556 1359 raft_consensus.cc:491] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:38.338140 1359 raft_consensus.cc:3058] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.340948 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:38.341739 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:38.342500 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:38.343091 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:38.344187 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:38.345525 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:38.346688 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:38.347198 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 4b7416ad6a244d978457e529cbc789de (DEFAULT_TABLE table=default.Table_06e661ea03304faaaa16113c730a6d91 [id=60538f85a9b84a1190ef7403747476fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.347285 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet d5cd316e6f4447be958611ab07ed31fc (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:38.347692 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b7416ad6a244d978457e529cbc789de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.348184 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:38.332582 339 tablet_bootstrap.cc:492] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.335636 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 3bfad68d1f20486997ea1b0a8d97206b (DEFAULT_TABLE table=default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.348966 339 ts_tablet_manager.cc:1397] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.002s
I20250623 08:04:38.338685 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 1494f83d13cd49c7a804593f45f97cc6 (DEFAULT_TABLE table=default.Table_d87c4052e62d4c1c87fc25cff314319b [id=58d5dec0c723470a9a7e56ce37dc71d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.339067 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 4b7416ad6a244d978457e529cbc789de (DEFAULT_TABLE table=default.Table_06e661ea03304faaaa16113c730a6d91 [id=60538f85a9b84a1190ef7403747476fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.349503 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bfad68d1f20486997ea1b0a8d97206b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.349867 339 raft_consensus.cc:357] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.350032 339 raft_consensus.cc:383] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.350090 339 raft_consensus.cc:738] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.350147 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1494f83d13cd49c7a804593f45f97cc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.350287 339 consensus_queue.cc:260] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.350802 339 ts_tablet_manager.cc:1428] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:38.351055 339 tablet_bootstrap.cc:492] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.352998 339 tablet_bootstrap.cc:654] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.354923 981 log.cc:1199] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/38ecd4c243b840f4b66d8e01078dca64
I20250623 08:04:38.355274 981 ts_tablet_manager.cc:1939] T 38ecd4c243b840f4b66d8e01078dca64 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:38.355309 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b7416ad6a244d978457e529cbc789de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.356014 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 166) took 3694 ms. Trace:
I20250623 08:04:38.356084 981 rpcz_store.cc:276] 0623 08:04:34.661275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.661314 (+ 39us) service_pool.cc:225] Handling call
0623 08:04:38.240326 (+3579012us) ts_tablet_manager.cc:1145] Deleting tablet 38ecd4c243b840f4b66d8e01078dca64
0623 08:04:38.240338 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.280788 (+ 40450us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.330438 (+ 49650us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.355994 (+ 25556us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3578811}
I20250623 08:04:38.356428 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 167) took 3694 ms. Trace:
I20250623 08:04:38.356489 981 rpcz_store.cc:276] 0623 08:04:34.661711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.661754 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:38.356324 (+3694570us) ts_tablet_manager.cc:1145] Deleting tablet 2060a643d0da47efa868f52bdb331b1f
0623 08:04:38.356333 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.356414 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3694405}
I20250623 08:04:38.356716 981 tablet_replica.cc:331] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:38.356869 981 raft_consensus.cc:2241] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.356982 981 raft_consensus.cc:2270] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.357491 981 ts_tablet_manager.cc:1905] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.360322 1359 raft_consensus.cc:513] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.362984 32453 leader_election.cc:304] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.363694 979 ts_tablet_manager.cc:1918] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.365175 1507 raft_consensus.cc:2802] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:38.365499 1507 raft_consensus.cc:491] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:38.366082 979 log.cc:1199] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/26f4a1b472e848ccb9a1ec71c1dd3516
I20250623 08:04:38.366277 1507 raft_consensus.cc:3058] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.366580 979 ts_tablet_manager.cc:1939] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.366948 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 215) took 2346 ms. Trace:
I20250623 08:04:38.367030 979 rpcz_store.cc:276] 0623 08:04:36.020591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.020660 (+ 69us) service_pool.cc:225] Handling call
0623 08:04:38.278179 (+2257519us) ts_tablet_manager.cc:1145] Deleting tablet 26f4a1b472e848ccb9a1ec71c1dd3516
0623 08:04:38.278188 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.347902 (+ 69714us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.351025 (+ 3123us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.366931 (+ 15906us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2257313}
I20250623 08:04:38.369855 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 214) took 2349 ms. Trace:
I20250623 08:04:38.369938 979 rpcz_store.cc:276] 0623 08:04:36.020406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.020657 (+ 251us) service_pool.cc:225] Handling call
0623 08:04:38.367313 (+2346656us) ts_tablet_manager.cc:1145] Deleting tablet 89bb7ea31b4d4ce7a3c063853142fbb2
0623 08:04:38.367325 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.369839 (+ 2514us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346358}
I20250623 08:04:38.369396 1507 raft_consensus.cc:513] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.370791 979 tablet_replica.cc:331] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.371011 979 raft_consensus.cc:2241] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:38.371333 979 raft_consensus.cc:2270] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.371932 979 ts_tablet_manager.cc:1905] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.371960 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 26f4a1b472e848ccb9a1ec71c1dd3516 (table default.Table_61ccd9db16cb4618874bff73413fd485 [id=8d6f6785bcf24b719419015368a05f68]) successfully deleted
I20250623 08:04:38.371390 1507 leader_election.cc:290] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:38.373363 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89bb7ea31b4d4ce7a3c063853142fbb2 already in progress: creating tablet
I20250623 08:04:38.378187 979 ts_tablet_manager.cc:1918] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.378345 979 log.cc:1199] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/2f62ec83c88e43438b963e740d2c7004
W20250623 08:04:38.379160 32314 leader_election.cc:343] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 70f9d21b8c144f9cada9bdbe0fbb1538
I20250623 08:04:38.379585 32314 leader_election.cc:304] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.380003 1332 raft_consensus.cc:2747] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:38.381012 32314 leader_election.cc:343] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 70f9d21b8c144f9cada9bdbe0fbb1538
I20250623 08:04:38.381271 32314 leader_election.cc:304] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.381670 1332 raft_consensus.cc:2747] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.382105 979 ts_tablet_manager.cc:1939] T 2f62ec83c88e43438b963e740d2c7004 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.382563 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 217) took 2259 ms. Trace:
I20250623 08:04:38.382772 979 rpcz_store.cc:276] 0623 08:04:36.122865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.123657 (+ 792us) service_pool.cc:225] Handling call
0623 08:04:38.370409 (+2246752us) ts_tablet_manager.cc:1145] Deleting tablet 2f62ec83c88e43438b963e740d2c7004
0623 08:04:38.370418 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.375177 (+ 4759us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.378066 (+ 2889us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.382499 (+ 4433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2246494}
I20250623 08:04:38.383409 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 2f62ec83c88e43438b963e740d2c7004 (table default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac [id=65c4e79e095b41b2b2a42e9858485423]) successfully deleted
I20250623 08:04:38.383714 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 38ecd4c243b840f4b66d8e01078dca64 (table default.Table_d895ba420734421e8b8f074848f4e389 [id=4116d3ec72a14031a9ac033b29ff2fe9]) successfully deleted
W20250623 08:04:38.384032 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 2060a643d0da47efa868f52bdb331b1f because the tablet was not found. No further retry: Not found: Tablet not found: 2060a643d0da47efa868f52bdb331b1f
I20250623 08:04:38.384300 981 ts_tablet_manager.cc:1918] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:38.384438 981 log.cc:1199] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/4cb66c7a273a412db3ae77af9e30d036
I20250623 08:04:38.384786 981 ts_tablet_manager.cc:1939] T 4cb66c7a273a412db3ae77af9e30d036 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:38.385437 32453 leader_election.cc:304] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.385771 1447 raft_consensus.cc:2747] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.386837 342 tablet_bootstrap.cc:492] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:38.386912 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 218) took 2243 ms. Trace:
I20250623 08:04:38.387085 342 ts_tablet_manager.cc:1397] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.001s
I20250623 08:04:38.387012 979 rpcz_store.cc:276] 0623 08:04:36.143136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.143697 (+ 561us) service_pool.cc:225] Handling call
0623 08:04:38.383231 (+2239534us) ts_tablet_manager.cc:1145] Deleting tablet 70f9d21b8c144f9cada9bdbe0fbb1538
0623 08:04:38.383239 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.386853 (+ 3614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239340}
I20250623 08:04:38.387358 979 tablet_replica.cc:331] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.387585 979 raft_consensus.cc:2241] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.387789 979 raft_consensus.cc:2270] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:38.388070 32453 consensus_peers.cc:487] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6ed4cb252b1452087f8ee53ec78d985. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:38.388643 32453 leader_election.cc:304] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.388769 979 ts_tablet_manager.cc:1905] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:38.388934 32453 raft_consensus.cc:2684] T ecb422550efd40cf82bc92fca5c25eda P 59f7c614254f42009306b13337f06576: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:38.389495 32453 leader_election.cc:304] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.389854 1447 raft_consensus.cc:2802] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:38.389958 1447 raft_consensus.cc:491] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:38.390069 1447 raft_consensus.cc:3058] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:38.390949 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 because the tablet was not found. No further retry: Not found: Tablet not found: 70f9d21b8c144f9cada9bdbe0fbb1538
I20250623 08:04:38.391511 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 168) took 3729 ms. Trace:
I20250623 08:04:38.391602 981 rpcz_store.cc:276] 0623 08:04:34.662102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.662166 (+ 64us) service_pool.cc:225] Handling call
0623 08:04:38.356675 (+3694509us) ts_tablet_manager.cc:1145] Deleting tablet 4cb66c7a273a412db3ae77af9e30d036
0623 08:04:38.356683 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.372480 (+ 15797us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.384185 (+ 11705us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.391490 (+ 7305us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3694300}
I20250623 08:04:38.392007 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 169) took 3729 ms. Trace:
I20250623 08:04:38.392088 981 rpcz_store.cc:276] 0623 08:04:34.662565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.782904 (+120339us) service_pool.cc:225] Handling call
0623 08:04:38.391881 (+3608977us) ts_tablet_manager.cc:1145] Deleting tablet 92005abc61dd40ef9da8124cadc29b6a
0623 08:04:38.391895 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.391991 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3608749}
I20250623 08:04:38.392481 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 170) took 3609 ms. Trace:
I20250623 08:04:38.392570 981 rpcz_store.cc:276] 0623 08:04:34.783362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.783812 (+ 450us) service_pool.cc:225] Handling call
0623 08:04:38.392344 (+3608532us) ts_tablet_manager.cc:1145] Deleting tablet 279624b8bb834b95a4f08045968ebfb4
0623 08:04:38.392353 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.392467 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3608336}
I20250623 08:04:38.392510 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:38.392906 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 171) took 3608 ms. Trace:
I20250623 08:04:38.392990 981 rpcz_store.cc:276] 0623 08:04:34.784184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.792857 (+ 8673us) service_pool.cc:225] Handling call
0623 08:04:38.392826 (+3599969us) ts_tablet_manager.cc:1145] Deleting tablet 19536c16ff32460e9db571e4ce411bab
0623 08:04:38.392836 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.392893 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3599693}
I20250623 08:04:38.393272 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 172) took 3599 ms. Trace:
I20250623 08:04:38.393306 1447 raft_consensus.cc:513] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.393345 981 rpcz_store.cc:276] 0623 08:04:34.793426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.793478 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:38.393192 (+3599714us) ts_tablet_manager.cc:1145] Deleting tablet 1e47b10f46b8402e94354ebca573c194
0623 08:04:38.393205 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.393260 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3599470}
I20250623 08:04:38.393517 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:38.393676 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 173) took 3599 ms. Trace:
I20250623 08:04:38.393764 981 rpcz_store.cc:276] 0623 08:04:34.793936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.796198 (+ 2262us) service_pool.cc:225] Handling call
0623 08:04:38.393589 (+3597391us) ts_tablet_manager.cc:1145] Deleting tablet 175ee90133994154b7e4b58b0770db3b
0623 08:04:38.393598 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.393656 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3597204}
I20250623 08:04:38.394052 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 174) took 3597 ms. Trace:
I20250623 08:04:38.394114 1447 leader_election.cc:290] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.394376 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:38.331892 952 ts_tablet_manager.cc:1905] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.395646 32056 catalog_manager.cc:5582] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:38.395157 339 tablet_bootstrap.cc:492] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.396044 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 025365d2fa0a43dd983d395a05cc2af1 (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:38.396090 339 ts_tablet_manager.cc:1397] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.013s
I20250623 08:04:38.396432 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:38.397034 339 raft_consensus.cc:357] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.397292 339 raft_consensus.cc:383] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.397394 339 raft_consensus.cc:738] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.397702 339 consensus_queue.cc:260] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.398236 343 tablet_bootstrap.cc:492] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.398478 343 ts_tablet_manager.cc:1397] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.152s user 0.028s sys 0.057s
I20250623 08:04:38.398502 339 ts_tablet_manager.cc:1428] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:38.399128 339 tablet_bootstrap.cc:492] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.399329 343 raft_consensus.cc:357] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.399585 343 raft_consensus.cc:383] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.398379 342 raft_consensus.cc:357] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.399704 343 raft_consensus.cc:738] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.399722 342 raft_consensus.cc:383] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.399900 342 raft_consensus.cc:738] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.400034 343 consensus_queue.cc:260] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.400583 343 ts_tablet_manager.cc:1428] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:38.400521 342 consensus_queue.cc:260] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.400949 343 tablet_bootstrap.cc:492] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:38.401216 339 tablet_bootstrap.cc:654] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.401520 32453 leader_election.cc:304] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.402136 981 rpcz_store.cc:276] 0623 08:04:34.796609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.796666 (+ 57us) service_pool.cc:225] Handling call
0623 08:04:38.393964 (+3597298us) ts_tablet_manager.cc:1145] Deleting tablet 5629d11c44e241d598f3e519b2fe1a44
0623 08:04:38.393974 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.394038 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3597107}
I20250623 08:04:38.402210 32453 leader_election.cc:304] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.402611 343 tablet_bootstrap.cc:654] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.402654 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 175) took 3605 ms. Trace:
I20250623 08:04:38.402737 1627 raft_consensus.cc:2802] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:38.402874 1627 raft_consensus.cc:695] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:38.402810 981 rpcz_store.cc:276] 0623 08:04:34.797075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.797115 (+ 40us) service_pool.cc:225] Handling call
0623 08:04:38.402528 (+3605413us) ts_tablet_manager.cc:1145] Deleting tablet f3c850021e914530bb106ab7166be7c9
0623 08:04:38.402540 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.402635 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3605250}
I20250623 08:04:38.403144 981 tablet_replica.cc:331] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:38.403117 1627 consensus_queue.cc:237] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.403390 981 raft_consensus.cc:2241] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.403573 981 raft_consensus.cc:2270] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.396677 32056 catalog_manager.cc:5582] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:38.404356 32056 catalog_manager.cc:5582] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:38.405109 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 5 got 0
I20250623 08:04:38.405550 32056 catalog_manager.cc:5582] T b19e5f2359a84c7ea02a0b25240a54cb P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:38.405961 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:38.406167 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:38.406491 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:38.406877 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:38.407016 32056 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:38.407617 1507 raft_consensus.cc:2802] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:38.408078 1507 raft_consensus.cc:695] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:38.408736 1507 consensus_queue.cc:237] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.410960 981 ts_tablet_manager.cc:1905] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.411057 32055 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f97f14cbaffb46d681d8b11c0213cae4
I20250623 08:04:38.411638 32055 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 307) took 1202 ms. Trace:
I20250623 08:04:38.411893 979 ts_tablet_manager.cc:1918] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.411760 32055 rpcz_store.cc:276] 0623 08:04:37.209247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.639453 (+430206us) service_pool.cc:225] Handling call
0623 08:04:37.640083 (+ 630us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:37.640280 (+ 197us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:37.640358 (+ 78us) catalog_manager.cc:2799] Locking tablets
0623 08:04:37.828164 (+187806us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:37.911015 (+ 82851us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:37.911100 (+ 85us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:38.410980 (+499880us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:38.411587 (+ 607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:37.842176 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:37.842295 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:37.842300 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:37.842303 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:37.842433 (+ 130us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:37.842442 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:37.842446 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:37.842538 (+ 92us) tablet.cc:699] Row locks acquired
0623 08:04:37.842541 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:37.842569 (+ 28us) write_op.cc:270] Start()
0623 08:04:37.842594 (+ 25us) write_op.cc:276] Timestamp: P: 1750665877842553 usec, L: 0
0623 08:04:37.842597 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:37.842743 (+ 146us) log.cc:844] Serialized 725 byte log entry
0623 08:04:37.910318 (+ 67575us) op_driver.cc:464] REPLICATION: finished
0623 08:04:37.910451 (+ 133us) write_op.cc:301] APPLY: starting
0623 08:04:37.910469 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:37.910513 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:37.910536 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:37.910601 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:37.910650 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:37.910656 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:37.910738 (+ 82us) log.cc:844] Serialized 29 byte log entry
0623 08:04:37.910824 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:37.910844 (+ 20us) write_op.cc:454] Released schema lock
0623 08:04:37.910851 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:37.910917 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:37.910920 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"apply.run_cpu_time_us":461,"apply.run_wall_time_us":461,"num_ops":2,"prepare.queue_time_us":13269,"prepare.run_cpu_time_us":646,"prepare.run_wall_time_us":647,"replication_time_us":67687}]]}
I20250623 08:04:38.412699 979 log.cc:1199] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/a414144276f6484ea4690204752e2142
W20250623 08:04:38.413030 1353 log.cc:927] Time spent T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Append to log took a long time: real 0.070s user 0.003s sys 0.000s
I20250623 08:04:38.413288 979 ts_tablet_manager.cc:1939] T a414144276f6484ea4690204752e2142 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.414110 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 219) took 2270 ms. Trace:
I20250623 08:04:38.414624 979 rpcz_store.cc:276] 0623 08:04:36.143292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.143944 (+ 652us) service_pool.cc:225] Handling call
0623 08:04:38.387260 (+2243316us) ts_tablet_manager.cc:1145] Deleting tablet a414144276f6484ea4690204752e2142
0623 08:04:38.387271 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.391950 (+ 4679us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.411730 (+ 19780us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.414078 (+ 2348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2243159}
I20250623 08:04:38.414783 1332 raft_consensus.cc:491] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.414911 1332 raft_consensus.cc:513] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.415537 1332 leader_election.cc:290] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.415670 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet a414144276f6484ea4690204752e2142 (table default.Table_a4ca66fa08cc441ab5571f9968f30914 [id=c3c5e1869ba745b9b2ab29742806f241]) successfully deleted
I20250623 08:04:38.416018 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 221) took 2272 ms. Trace:
I20250623 08:04:38.416100 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet f97f14cbaffb46d681d8b11c0213cae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:38.416268 979 rpcz_store.cc:276] 0623 08:04:36.143516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.144402 (+ 886us) service_pool.cc:225] Handling call
0623 08:04:38.415424 (+2271022us) ts_tablet_manager.cc:1145] Deleting tablet 38ecd4c243b840f4b66d8e01078dca64
0623 08:04:38.415433 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.415514 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2270876}
W20250623 08:04:38.416966 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 38ecd4c243b840f4b66d8e01078dca64 because the tablet was not found. No further retry: Not found: Tablet not found: 38ecd4c243b840f4b66d8e01078dca64
I20250623 08:04:38.417438 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 222) took 2273 ms. Trace:
I20250623 08:04:38.417888 979 rpcz_store.cc:276] 0623 08:04:36.143647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.144576 (+ 929us) service_pool.cc:225] Handling call
0623 08:04:38.417302 (+2272726us) ts_tablet_manager.cc:1145] Deleting tablet 5fdc9e53da264a5ca1f70e1846c906f0
0623 08:04:38.417314 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.417402 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2272585}
W20250623 08:04:38.418545 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 5fdc9e53da264a5ca1f70e1846c906f0 because the tablet was not found. No further retry: Not found: Tablet not found: 5fdc9e53da264a5ca1f70e1846c906f0
I20250623 08:04:38.418577 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 223) took 2235 ms. Trace:
I20250623 08:04:38.418681 979 rpcz_store.cc:276] 0623 08:04:36.182717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.182788 (+ 71us) service_pool.cc:225] Handling call
0623 08:04:38.418462 (+2235674us) ts_tablet_manager.cc:1145] Deleting tablet 16331ba6a34942a5a1b4ff4ac7bac55e
0623 08:04:38.418470 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.418547 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2235476}
I20250623 08:04:38.419318 342 ts_tablet_manager.cc:1428] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
W20250623 08:04:38.419598 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 16331ba6a34942a5a1b4ff4ac7bac55e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16331ba6a34942a5a1b4ff4ac7bac55e already in progress: creating tablet
I20250623 08:04:38.420018 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 225) took 2203 ms. Trace:
I20250623 08:04:38.420423 979 rpcz_store.cc:276] 0623 08:04:36.216252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.216310 (+ 58us) service_pool.cc:225] Handling call
0623 08:04:38.419893 (+2203583us) ts_tablet_manager.cc:1145] Deleting tablet b84b8b8713774ab4896842feb9fb075e
0623 08:04:38.419903 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.419988 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2203305}
W20250623 08:04:38.421022 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b84b8b8713774ab4896842feb9fb075e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84b8b8713774ab4896842feb9fb075e already in progress: creating tablet
I20250623 08:04:38.421437 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 226) took 2184 ms. Trace:
I20250623 08:04:38.421844 979 rpcz_store.cc:276] 0623 08:04:36.236812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.240784 (+ 3972us) service_pool.cc:225] Handling call
0623 08:04:38.421296 (+2180512us) ts_tablet_manager.cc:1145] Deleting tablet 9bfa787f828b411ab68d81bb896f7fb6
0623 08:04:38.421303 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.421402 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2119222}
W20250623 08:04:38.422499 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9bfa787f828b411ab68d81bb896f7fb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bfa787f828b411ab68d81bb896f7fb6 already in progress: creating tablet
I20250623 08:04:38.422946 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 234) took 2082 ms. Trace:
I20250623 08:04:38.423375 979 rpcz_store.cc:276] 0623 08:04:36.340319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.340920 (+ 601us) service_pool.cc:225] Handling call
0623 08:04:38.422822 (+2081902us) ts_tablet_manager.cc:1145] Deleting tablet f5f58f55070344ff98a3808f70e0c755
0623 08:04:38.422830 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.422916 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2081598}
I20250623 08:04:38.426512 981 ts_tablet_manager.cc:1918] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.427011 981 log.cc:1199] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/a9c8d480b8ba49a1be140b0e0a79f4e4
W20250623 08:04:38.427130 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f5f58f55070344ff98a3808f70e0c755 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5f58f55070344ff98a3808f70e0c755 already in progress: creating tablet
I20250623 08:04:38.427577 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 240) took 2002 ms. Trace:
I20250623 08:04:38.428021 979 rpcz_store.cc:276] 0623 08:04:36.425040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.435901 (+ 10861us) service_pool.cc:225] Handling call
0623 08:04:38.427446 (+1991545us) ts_tablet_manager.cc:1145] Deleting tablet b61b02cf2904404f8c0ffef7d268fefc
0623 08:04:38.427455 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.427546 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1991292}
I20250623 08:04:38.428678 979 tablet_replica.cc:331] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.428809 981 ts_tablet_manager.cc:1939] T a9c8d480b8ba49a1be140b0e0a79f4e4 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:38.428983 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b61b02cf2904404f8c0ffef7d268fefc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b61b02cf2904404f8c0ffef7d268fefc already in progress: creating tablet
I20250623 08:04:38.429114 979 raft_consensus.cc:2241] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.429572 979 raft_consensus.cc:2270] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.430424 979 ts_tablet_manager.cc:1905] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.431087 342 tablet_bootstrap.cc:492] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:38.432094 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.432302 32650 raft_consensus.cc:2391] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:38.433871 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 4cb66c7a273a412db3ae77af9e30d036 (table default.Table_347f265adebd45bdb27956d1affc3e89 [id=e7c90b63c16a4cca9365fd4733ba1296]) successfully deleted
W20250623 08:04:38.434144 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 92005abc61dd40ef9da8124cadc29b6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92005abc61dd40ef9da8124cadc29b6a already in progress: creating tablet
W20250623 08:04:38.434451 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 279624b8bb834b95a4f08045968ebfb4 because the tablet was not found. No further retry: Not found: Tablet not found: 279624b8bb834b95a4f08045968ebfb4
W20250623 08:04:38.434684 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 19536c16ff32460e9db571e4ce411bab because the tablet was not found. No further retry: Not found: Tablet not found: 19536c16ff32460e9db571e4ce411bab
W20250623 08:04:38.434962 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 1e47b10f46b8402e94354ebca573c194 because the tablet was not found. No further retry: Not found: Tablet not found: 1e47b10f46b8402e94354ebca573c194
W20250623 08:04:38.435173 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 175ee90133994154b7e4b58b0770db3b because the tablet was not found. No further retry: Not found: Tablet not found: 175ee90133994154b7e4b58b0770db3b
W20250623 08:04:38.435412 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 5629d11c44e241d598f3e519b2fe1a44 because the tablet was not found. No further retry: Not found: Tablet not found: 5629d11c44e241d598f3e519b2fe1a44
W20250623 08:04:38.435671 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet f3c850021e914530bb106ab7166be7c9 because the tablet was not found. No further retry: Not found: Tablet not found: f3c850021e914530bb106ab7166be7c9
I20250623 08:04:38.441510 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 025365d2fa0a43dd983d395a05cc2af1
I20250623 08:04:38.441968 32053 catalog_manager.cc:6652] ReplaceTablet: tablet 025365d2fa0a43dd983d395a05cc2af1 deleted and replaced by tablet a375f8ea383b44a9acd04fc2171d34ae
I20250623 08:04:38.442090 32053 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36116 (request call id 320) took 1015 ms. Trace:
I20250623 08:04:38.442176 32053 rpcz_store.cc:276] 0623 08:04:37.426734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.770818 (+344084us) service_pool.cc:225] Handling call
0623 08:04:37.849887 (+ 79069us) spinlock_profiling.cc:243] Waited 50.9 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d080605b 00007f19c764f5d7 00007f19c764f305 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:38.442063 (+592176us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:38.437262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:38.437371 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:38.437377 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:38.437379 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:38.437536 (+ 157us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:38.437548 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:38.437553 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:38.437632 (+ 79us) tablet.cc:699] Row locks acquired
0623 08:04:38.437635 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:38.437668 (+ 33us) write_op.cc:270] Start()
0623 08:04:38.437699 (+ 31us) write_op.cc:276] Timestamp: P: 1750665878437644 usec, L: 0
0623 08:04:38.437703 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:38.437871 (+ 168us) log.cc:844] Serialized 656 byte log entry
0623 08:04:38.440564 (+ 2693us) op_driver.cc:464] REPLICATION: finished
0623 08:04:38.440660 (+ 96us) write_op.cc:301] APPLY: starting
0623 08:04:38.440674 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:38.440713 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:38.440745 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:38.440838 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:38.440887 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:38.440895 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:38.440958 (+ 63us) log.cc:844] Serialized 29 byte log entry
0623 08:04:38.441007 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:38.441040 (+ 33us) write_op.cc:454] Released schema lock
0623 08:04:38.441047 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:38.441114 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:38.441118 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":117009280,"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":442,"apply.run_wall_time_us":442,"num_ops":2,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":696,"prepare.run_wall_time_us":696,"raft.queue_time_us":401,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":196,"replication_time_us":2833}]]}
I20250623 08:04:38.443032 32053 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_1da3c8ca68084386b7f0d95a6f85927a" } new_table_name: "default.Table_161ad5ddd1cb40e0989e5c237a205db1" modify_external_catalogs: true
I20250623 08:04:38.446924 1332 raft_consensus.cc:491] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.447067 1332 raft_consensus.cc:513] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.447671 1332 leader_election.cc:290] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.452278 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 025365d2fa0a43dd983d395a05cc2af1 with delete_type TABLET_DATA_DELETED (replaced by tablet a375f8ea383b44a9acd04fc2171d34ae) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:38.432124 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.454988 32518 raft_consensus.cc:2391] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:38.450314 1575 consensus_queue.cc:1035] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:38.459901 32346 tablet_service.cc:1515] Processing DeleteTablet for tablet f97f14cbaffb46d681d8b11c0213cae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.460122 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 025365d2fa0a43dd983d395a05cc2af1 with delete_type TABLET_DATA_DELETED (replaced by tablet a375f8ea383b44a9acd04fc2171d34ae) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.460659 32314 leader_election.cc:304] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.450587 32453 leader_election.cc:304] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:38.462617 1575 raft_consensus.cc:2802] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:38.462702 1575 raft_consensus.cc:491] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:38.462770 1575 raft_consensus.cc:3058] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.465418 1575 raft_consensus.cc:513] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.466012 1575 leader_election.cc:290] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:38.467160 32453 consensus_peers.cc:487] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ffafd10f26740c9a5b6c9bdc3bb23a0. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:38.467743 32483 tablet_service.cc:1468] Processing CreateTablet for tablet 38b8cfcbce6a472ab2daae28226cf766 (DEFAULT_TABLE table=default.Table_b161fc2726dc4831988f5ac03028b089 [id=c8effd1e07264a5984e51f380a700652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.468119 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b8cfcbce6a472ab2daae28226cf766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.454028 1359 leader_election.cc:290] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.485286 1634 raft_consensus.cc:491] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.485420 1634 raft_consensus.cc:513] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.486024 1634 leader_election.cc:290] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.489706 32483 tablet_service.cc:1468] Processing CreateTablet for tablet ee82db25aa544a21a1419e081196f179 (DEFAULT_TABLE table=default.Table_c1548afc4cff490a94e8a03d5fea9339 [id=89148e37f2454772a74b0c88e4164f0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.490147 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee82db25aa544a21a1419e081196f179. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.500646 979 ts_tablet_manager.cc:1918] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.502629 979 log.cc:1199] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/48da1e4759bb4fea87a45ad8a428b6e7
I20250623 08:04:38.503196 979 ts_tablet_manager.cc:1939] T 48da1e4759bb4fea87a45ad8a428b6e7 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.503954 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 239) took 2079 ms. Trace:
I20250623 08:04:38.504434 979 rpcz_store.cc:276] 0623 08:04:36.424922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.439962 (+ 15040us) service_pool.cc:225] Handling call
0623 08:04:38.428604 (+1988642us) ts_tablet_manager.cc:1145] Deleting tablet 48da1e4759bb4fea87a45ad8a428b6e7
0623 08:04:38.428615 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.485263 (+ 56648us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.500548 (+ 15285us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.503936 (+ 3388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1988355}
I20250623 08:04:38.505077 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 242) took 1976 ms. Trace:
I20250623 08:04:38.505278 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 48da1e4759bb4fea87a45ad8a428b6e7 (table default.Table_c382db2a0c144ec4abd51b3e927e0a17 [id=461d38c05dc24245b0c59cb73b42a394]) successfully deleted
I20250623 08:04:38.505470 979 rpcz_store.cc:276] 0623 08:04:36.528150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.528203 (+ 53us) service_pool.cc:225] Handling call
0623 08:04:38.504973 (+1976770us) ts_tablet_manager.cc:1145] Deleting tablet a9c8d480b8ba49a1be140b0e0a79f4e4
0623 08:04:38.504984 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.505059 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1976573}
I20250623 08:04:38.505947 979 tablet_replica.cc:331] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
W20250623 08:04:38.506171 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 because the tablet was not found. No further retry: Not found: Tablet not found: a9c8d480b8ba49a1be140b0e0a79f4e4
I20250623 08:04:38.506307 979 raft_consensus.cc:2241] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.506680 979 raft_consensus.cc:2270] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.507380 979 ts_tablet_manager.cc:1905] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.514521 1506 raft_consensus.cc:491] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.514654 1506 raft_consensus.cc:513] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.515215 1506 leader_election.cc:290] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.519330 979 ts_tablet_manager.cc:1918] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:38.527151 979 log.cc:1199] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/9a0bbafe1ea34ab5ae1b3de25b50d4a3
I20250623 08:04:38.527667 979 ts_tablet_manager.cc:1939] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.528765 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 241) took 2000 ms. Trace:
I20250623 08:04:38.529642 979 rpcz_store.cc:276] 0623 08:04:36.528011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.528439 (+ 428us) service_pool.cc:225] Handling call
0623 08:04:38.505904 (+1977465us) ts_tablet_manager.cc:1145] Deleting tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3
0623 08:04:38.505913 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.516235 (+ 10322us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.519246 (+ 3011us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.528748 (+ 9502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1977306}
I20250623 08:04:38.530237 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 243) took 1970 ms. Trace:
I20250623 08:04:38.530678 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 (table default.Table_e7f8192c00524b5899152fa3f717b898 [id=f4277b270809454da8cfd8f3158ed6f8]) successfully deleted
I20250623 08:04:38.531963 32487 tablet_service.cc:1468] Processing CreateTablet for tablet a2a07188a4d340bc9caf010d5202e34d (DEFAULT_TABLE table=default.Table_1ced320c61ec4355bb2686a5c84a55eb [id=61c080721215406397c00aff27538c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.532325 32487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a07188a4d340bc9caf010d5202e34d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.534026 979 rpcz_store.cc:276] 0623 08:04:36.560136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.560220 (+ 84us) service_pool.cc:225] Handling call
0623 08:04:38.530125 (+1969905us) ts_tablet_manager.cc:1145] Deleting tablet b720fdea539f423ba2d2aa9f89b09ae9
0623 08:04:38.530134 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.530225 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1969726}
I20250623 08:04:38.534340 979 tablet_replica.cc:331] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.534518 979 raft_consensus.cc:2241] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:38.535494 979 raft_consensus.cc:2270] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:38.535820 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet b720fdea539f423ba2d2aa9f89b09ae9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b720fdea539f423ba2d2aa9f89b09ae9 already in progress: creating tablet
I20250623 08:04:38.536334 979 ts_tablet_manager.cc:1905] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.542018 952 ts_tablet_manager.cc:1918] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:38.468281 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 176) took 3670 ms. Trace:
I20250623 08:04:38.486721 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861e45cf546487d9812caa06e2704eb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:38.508646 1622 raft_consensus.cc:491] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.542402 1622 raft_consensus.cc:513] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.542272 981 rpcz_store.cc:276] 0623 08:04:34.797449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.797493 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:38.403087 (+3605594us) ts_tablet_manager.cc:1145] Deleting tablet a9c8d480b8ba49a1be140b0e0a79f4e4
0623 08:04:38.403099 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.417351 (+ 14252us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.426372 (+ 9021us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.468257 (+ 41885us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3605366}
I20250623 08:04:38.543409 1622 leader_election.cc:290] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.511184 1303 raft_consensus.cc:491] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.543689 1303 raft_consensus.cc:513] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.544214 1303 leader_election.cc:290] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.515947 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87648b11b10b45fc9cc8c998bbceae92" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:38.544837 32589 leader_election.cc:343] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:38.545096 32589 leader_election.cc:343] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.545492 1303 raft_consensus.cc:3058] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.548799 1303 raft_consensus.cc:2747] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.548848 979 ts_tablet_manager.cc:1918] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.548939 979 log.cc:1199] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b81263a9ee9d4da1ac721c3b2bd21248
I20250623 08:04:38.549264 979 ts_tablet_manager.cc:1939] T b81263a9ee9d4da1ac721c3b2bd21248 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:38.549575 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 88944.7us
I20250623 08:04:38.551674 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 244) took 1982 ms. Trace:
I20250623 08:04:38.552634 979 rpcz_store.cc:276] 0623 08:04:36.569006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.569059 (+ 53us) service_pool.cc:225] Handling call
0623 08:04:38.534283 (+1965224us) ts_tablet_manager.cc:1145] Deleting tablet b81263a9ee9d4da1ac721c3b2bd21248
0623 08:04:38.534302 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.540827 (+ 6525us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.548743 (+ 7916us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.551653 (+ 2910us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1964943}
I20250623 08:04:38.553241 979 tablet_replica.cc:331] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.553529 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b81263a9ee9d4da1ac721c3b2bd21248 (table default.Table_4e377b569121400d82096403060f9cda [id=49bfa95de97b443ba51225f84820fcf2]) successfully deleted
I20250623 08:04:38.553668 979 raft_consensus.cc:2241] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.553820 979 raft_consensus.cc:2270] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.558840 979 ts_tablet_manager.cc:1905] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.565979 32627 tablet_service.cc:1468] Processing CreateTablet for tablet e27da2a90c1b43deb3dc90834f4e312e (DEFAULT_TABLE table=default.Table_de7a3df7f0064a7bb7715c0c5dcc337e [id=5e7083acd09b41b2b78bd229dc7b1e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.566363 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27da2a90c1b43deb3dc90834f4e312e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.571467 1506 raft_consensus.cc:491] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.571643 32636 tablet_service.cc:1468] Processing CreateTablet for tablet e9a0c57133884896b03e9c0a8afb8bde (DEFAULT_TABLE table=default.Table_63b537810a3a4f478e4226cabee7004d [id=692d14d5e7224e37b0a78ac3171bfefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.571625 1506 raft_consensus.cc:513] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.571969 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9a0c57133884896b03e9c0a8afb8bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.572197 1506 leader_election.cc:290] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.575012 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 1c529e826fa84a919c018887620e4023 (DEFAULT_TABLE table=default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.575328 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c529e826fa84a919c018887620e4023. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.577734 32589 leader_election.cc:304] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.578105 1506 raft_consensus.cc:2747] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.578454 32630 tablet_service.cc:1468] Processing CreateTablet for tablet a3ad3d261a794d1184061c8e9b384693 (DEFAULT_TABLE table=default.Table_f663b3b4536b4a19a3c42d533d148127 [id=86475d6fed9c4f8b84556bcc1b1030be]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:38.579385 32589 leader_election.cc:343] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.579492 32589 leader_election.cc:304] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:38.579723 1506 raft_consensus.cc:2747] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.572880 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42c861e29e1e4e549021dae7255a856c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:38.580703 32589 leader_election.cc:343] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.580752 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21080ab069d74457a9f7fe3aabdb3c7c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.580960 32650 raft_consensus.cc:2391] T 21080ab069d74457a9f7fe3aabdb3c7c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:38.581262 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ad3d261a794d1184061c8e9b384693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.581696 32650 raft_consensus.cc:1273] T f7d69bcad8684d99a0b08eefb470ab01 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Refusing update from remote peer 7351aa31800245659edc9837e27daffd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 08:04:38.582216 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet a9c8d480b8ba49a1be140b0e0a79f4e4 (table default.Table_4550e74cc6e845719c1466d3acd1b723 [id=587d6896c7dd40838ce8ad01026379a8]) successfully deleted
I20250623 08:04:38.584632 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.584794 32650 raft_consensus.cc:2466] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:38.585641 1303 raft_consensus.cc:491] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.585798 1303 raft_consensus.cc:513] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:38.586180 32044 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 23928d64bcef4c06af81698e688b7f94 consensus state has no leader
I20250623 08:04:38.586407 1303 leader_election.cc:290] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.586851 1303 raft_consensus.cc:491] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.587007 1303 raft_consensus.cc:513] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.587563 1303 leader_election.cc:290] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.588450 32355 tablet_service.cc:1468] Processing CreateTablet for tablet e27da2a90c1b43deb3dc90834f4e312e (DEFAULT_TABLE table=default.Table_de7a3df7f0064a7bb7715c0c5dcc337e [id=5e7083acd09b41b2b78bd229dc7b1e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.589088 32342 tablet_service.cc:1468] Processing CreateTablet for tablet a3ad3d261a794d1184061c8e9b384693 (DEFAULT_TABLE table=default.Table_f663b3b4536b4a19a3c42d533d148127 [id=86475d6fed9c4f8b84556bcc1b1030be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.589085 32346 tablet_service.cc:1468] Processing CreateTablet for tablet aface2a39d764a6cb6e4965867ab7b38 (DEFAULT_TABLE table=default.Table_848fa16d96984425bfb59432e4b51723 [id=245da49183a0465d8159a9c92b729d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.589428 32348 tablet_service.cc:1468] Processing CreateTablet for tablet e9a0c57133884896b03e9c0a8afb8bde (DEFAULT_TABLE table=default.Table_63b537810a3a4f478e4226cabee7004d [id=692d14d5e7224e37b0a78ac3171bfefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.589979 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 1c529e826fa84a919c018887620e4023 (DEFAULT_TABLE table=default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.590422 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa604f500984e3b969e15592cdbed6f (DEFAULT_TABLE table=default.Table_54fe37774d62482296afe3191f2e8e88 [id=20b80e6a4e76480890e6d28402d54dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.590812 32650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d98654c29b9c4e75af6a213cafd4c2c9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.590922 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.591025 32650 raft_consensus.cc:2391] T d98654c29b9c4e75af6a213cafd4c2c9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:38.591087 32642 raft_consensus.cc:2391] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:38.592999 1303 consensus_queue.cc:1035] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:38.593737 32314 leader_election.cc:343] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: b80a91ae2c084020b5d3444b570b28f2
I20250623 08:04:38.594128 32314 leader_election.cc:304] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:38.598198 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 177) took 3800 ms. Trace:
I20250623 08:04:38.598318 981 rpcz_store.cc:276] 0623 08:04:34.798037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.798084 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:38.542778 (+3744694us) ts_tablet_manager.cc:1145] Deleting tablet 5fdc9e53da264a5ca1f70e1846c906f0
0623 08:04:38.598065 (+ 55287us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.598153 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3744453}
I20250623 08:04:38.598526 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27da2a90c1b43deb3dc90834f4e312e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.598687 981 tablet_replica.cc:331] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:38.598977 981 raft_consensus.cc:2241] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:38.599170 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa604f500984e3b969e15592cdbed6f (DEFAULT_TABLE table=default.Table_54fe37774d62482296afe3191f2e8e88 [id=20b80e6a4e76480890e6d28402d54dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.599299 981 raft_consensus.cc:2270] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.599509 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa604f500984e3b969e15592cdbed6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.600188 981 ts_tablet_manager.cc:1905] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.603120 32626 tablet_service.cc:1468] Processing CreateTablet for tablet aface2a39d764a6cb6e4965867ab7b38 (DEFAULT_TABLE table=default.Table_848fa16d96984425bfb59432e4b51723 [id=245da49183a0465d8159a9c92b729d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.603457 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aface2a39d764a6cb6e4965867ab7b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.604254 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aface2a39d764a6cb6e4965867ab7b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.604871 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa604f500984e3b969e15592cdbed6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.605365 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c529e826fa84a919c018887620e4023. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.605881 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9a0c57133884896b03e9c0a8afb8bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.611460 339 tablet_bootstrap.cc:492] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.611619 339 ts_tablet_manager.cc:1397] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.213s user 0.034s sys 0.088s
I20250623 08:04:38.612406 339 raft_consensus.cc:357] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.612586 339 raft_consensus.cc:383] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.612668 339 raft_consensus.cc:738] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.612908 339 consensus_queue.cc:260] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.613415 339 ts_tablet_manager.cc:1428] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:38.613549 32342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ad3d261a794d1184061c8e9b384693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.614168 981 ts_tablet_manager.cc:1918] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.614274 981 log.cc:1199] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/965f1f0a107b4653b6aac01e0d8e259a
I20250623 08:04:38.617007 1303 raft_consensus.cc:491] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.617151 1303 raft_consensus.cc:513] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.617714 1303 leader_election.cc:290] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.618618 981 ts_tablet_manager.cc:1939] T 965f1f0a107b4653b6aac01e0d8e259a P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:38.619880 343 tablet_bootstrap.cc:492] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.620028 343 ts_tablet_manager.cc:1397] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.219s user 0.027s sys 0.025s
I20250623 08:04:38.620733 343 raft_consensus.cc:357] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.622120 343 raft_consensus.cc:383] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.626801 1506 raft_consensus.cc:491] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.626928 1506 raft_consensus.cc:513] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.627467 1506 leader_election.cc:290] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.626814 343 raft_consensus.cc:738] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.624953 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 3d7773d6479742f79c38eca34018f94f (DEFAULT_TABLE table=default.Table_25f75684255a4b99a8aff38ee935aac5 [id=e85cbbd0d9db4945930b3fca419e3f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.628650 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7773d6479742f79c38eca34018f94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.628790 343 consensus_queue.cc:260] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.550312 342 tablet_bootstrap.cc:654] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.631328 952 log.cc:1199] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/c7fcf583064f4538913d9157e93068dd
I20250623 08:04:38.631762 952 ts_tablet_manager.cc:1939] T c7fcf583064f4538913d9157e93068dd P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:38.571038 32487 tablet_service.cc:1515] Processing DeleteTablet for tablet 886fc0fa39c54bddb25ac01f68d95805 with delete_type TABLET_DATA_DELETED (replaced by tablet ec4d50d5f9394afe97e3b43d65e6894e) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:38.571540 32496 tablet_service.cc:1468] Processing CreateTablet for tablet ba5913811e9343389c9ea0ec8b8c05ef (DEFAULT_TABLE table=default.Table_787415ae5937475f90f90000aa575d0e [id=4b27489168854629848531915104847a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.632438 32496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba5913811e9343389c9ea0ec8b8c05ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.632579 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 107) took 6173 ms. Trace:
I20250623 08:04:38.632645 952 rpcz_store.cc:276] 0623 08:04:32.459185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.459236 (+ 51us) service_pool.cc:225] Handling call
0623 08:04:38.278415 (+5819179us) ts_tablet_manager.cc:1145] Deleting tablet c7fcf583064f4538913d9157e93068dd
0623 08:04:38.278426 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.452202 (+173776us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.541925 (+ 89723us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.632561 (+ 90636us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5818956}
I20250623 08:04:38.632822 343 ts_tablet_manager.cc:1428] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250623 08:04:38.632943 952 tablet_replica.cc:331] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:38.633065 343 tablet_bootstrap.cc:492] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:38.633109 952 raft_consensus.cc:2241] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:38.633246 952 pending_rounds.cc:70] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:38.633277 952 pending_rounds.cc:77] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727426929065984 op_type: NO_OP noop_request { }
I20250623 08:04:38.633352 952 raft_consensus.cc:2887] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:38.633435 952 raft_consensus.cc:2270] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.633903 952 ts_tablet_manager.cc:1905] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.634310 343 tablet_bootstrap.cc:654] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.572041 32498 tablet_service.cc:1468] Processing CreateTablet for tablet c2c990297f554adbb2d97f6309758940 (DEFAULT_TABLE table=default.Table_3cbae8872bcd4af3a1a837940945fab6 [id=0ea2aee143054b39ae96e30c64a42a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.635002 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2c990297f554adbb2d97f6309758940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.635097 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet e9f8aa424bbb4aa8a907259270d46656 (table default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]) successfully deleted
I20250623 08:04:38.635337 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet ecb422550efd40cf82bc92fca5c25eda (table default.Table_68acbd516e8f4990a2f197ac131bc6ba [id=5545b396fc5746469e3a8908b0178a4c]) successfully deleted
I20250623 08:04:38.638655 1508 raft_consensus.cc:491] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.638888 1508 raft_consensus.cc:513] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.639413 1508 leader_election.cc:290] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.639554 1508 raft_consensus.cc:491] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.639676 1508 raft_consensus.cc:513] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.639974 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861e45cf546487d9812caa06e2704eb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.640184 32516 raft_consensus.cc:2466] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:38.640233 1508 leader_election.cc:290] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.640236 32511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87648b11b10b45fc9cc8c998bbceae92" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.572448 32489 tablet_service.cc:1468] Processing CreateTablet for tablet b4e01b430a064740b76e0bceec4f5546 (DEFAULT_TABLE table=default.Table_bcdbc5ed766f43ba91aaa4711f0b24f6 [id=7a2cd6d0739b48dca9ef91af18569181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.640810 339 tablet_bootstrap.cc:492] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.642678 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36c8714cfbd84dd38cf5c0f305ef6843" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:38.642979 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 48911.1us
W20250623 08:04:38.643249 32589 leader_election.cc:343] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.644711 32378 raft_consensus.cc:3058] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.644790 32373 raft_consensus.cc:1273] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Refusing update from remote peer 59f7c614254f42009306b13337f06576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 08:04:38.572960 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 1ae0a411a3e54c45b8d5c5b9b0eea762 (DEFAULT_TABLE table=default.Table_71f2f0414e0840be832096955576cb2f [id=a6918c5eb39a435ab4b035c1509916a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.645292 1506 raft_consensus.cc:491] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.645421 1506 raft_consensus.cc:513] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.645956 1506 leader_election.cc:290] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.646152 32379 raft_consensus.cc:3058] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.646361 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e602d391526d4b0eb0dbf5189afbe075" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.646528 32649 raft_consensus.cc:2466] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:38.646643 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa398354a9e44633857a75c0bbba08b3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:38.646780 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "724d662bcf754c74a8f3dff91ee258ae" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:38.646842 32374 raft_consensus.cc:2466] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:38.646895 32645 raft_consensus.cc:3058] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.647717 32589 leader_election.cc:304] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:38.648236 1506 raft_consensus.cc:2802] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:38.648301 1506 raft_consensus.cc:491] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:38.648353 1506 raft_consensus.cc:3058] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.649364 32645 raft_consensus.cc:2466] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:38.649621 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfd3e263464c4f2c8af9262acf211193" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:38.649772 32641 raft_consensus.cc:3058] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.651037 1332 raft_consensus.cc:2802] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:38.651139 1332 raft_consensus.cc:491] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:38.651232 1332 raft_consensus.cc:3058] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.652962 32641 raft_consensus.cc:2466] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:38.653780 1332 raft_consensus.cc:513] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.654265 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:38.654384 1332 leader_election.cc:290] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.654438 32648 raft_consensus.cc:2391] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
W20250623 08:04:38.655097 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 5fdc9e53da264a5ca1f70e1846c906f0 because the tablet was not found. No further retry: Not found: Tablet not found: 5fdc9e53da264a5ca1f70e1846c906f0
I20250623 08:04:38.655148 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:38.655267 32653 raft_consensus.cc:3058] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.655504 1506 raft_consensus.cc:513] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.656347 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa398354a9e44633857a75c0bbba08b3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:38.656569 32379 raft_consensus.cc:3058] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.657006 1506 leader_election.cc:290] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.658062 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ea63263b9b466e922169eecafab747" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:38.658151 32653 raft_consensus.cc:2466] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:38.658774 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e602d391526d4b0eb0dbf5189afbe075" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:38.659675 32379 raft_consensus.cc:2466] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:38.659797 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c33fa0a7ff34d2985c5e07fcf88eaa1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.573598 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 9bb99da3b8fa4dc6aa7f11a829d3f2d2 (DEFAULT_TABLE table=default.Table_3980cd736bd342f4bcd5cf289c1c8c25 [id=47a9c2f4140743c3a357032721181f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.660202 32589 leader_election.cc:304] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:38.661168 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "724d662bcf754c74a8f3dff91ee258ae" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:38.661336 32376 raft_consensus.cc:3058] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.574139 32484 tablet_service.cc:1468] Processing CreateTablet for tablet 1cee3de5953244f98428747bea912f77 (DEFAULT_TABLE table=default.Table_969668212d4f422ca8f2f34266315161 [id=eb6d9c5745e649978f87c97627ace9bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.574662 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 3bfad68d1f20486997ea1b0a8d97206b (DEFAULT_TABLE table=default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.575209 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 1494f83d13cd49c7a804593f45f97cc6 (DEFAULT_TABLE table=default.Table_d87c4052e62d4c1c87fc25cff314319b [id=58d5dec0c723470a9a7e56ce37dc71d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.575850 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 4b7416ad6a244d978457e529cbc789de (DEFAULT_TABLE table=default.Table_06e661ea03304faaaa16113c730a6d91 [id=60538f85a9b84a1190ef7403747476fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.576292 32488 tablet_service.cc:1515] Processing DeleteTablet for tablet f97f14cbaffb46d681d8b11c0213cae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:38.576601 32490 tablet_service.cc:1515] Processing DeleteTablet for tablet 025365d2fa0a43dd983d395a05cc2af1 with delete_type TABLET_DATA_DELETED (replaced by tablet a375f8ea383b44a9acd04fc2171d34ae) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:38.577106 32493 tablet_service.cc:1468] Processing CreateTablet for tablet e27da2a90c1b43deb3dc90834f4e312e (DEFAULT_TABLE table=default.Table_de7a3df7f0064a7bb7715c0c5dcc337e [id=5e7083acd09b41b2b78bd229dc7b1e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.577203 32497 tablet_service.cc:1468] Processing CreateTablet for tablet aface2a39d764a6cb6e4965867ab7b38 (DEFAULT_TABLE table=default.Table_848fa16d96984425bfb59432e4b51723 [id=245da49183a0465d8159a9c92b729d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.577517 32482 tablet_service.cc:1468] Processing CreateTablet for tablet e9a0c57133884896b03e9c0a8afb8bde (DEFAULT_TABLE table=default.Table_63b537810a3a4f478e4226cabee7004d [id=692d14d5e7224e37b0a78ac3171bfefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.579224 32481 tablet_service.cc:1468] Processing CreateTablet for tablet 1c529e826fa84a919c018887620e4023 (DEFAULT_TABLE table=default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.579255 32495 tablet_service.cc:1468] Processing CreateTablet for tablet e41b15c89a744a698ca5eee2f143b4bc (DEFAULT_TABLE table=default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.579562 32491 tablet_service.cc:1468] Processing CreateTablet for tablet 8f8f8a132e6245f9a04b5bb6f2253c76 (DEFAULT_TABLE table=default.Table_c748f600d56f4374bdb31ad502ec7557 [id=3b342fcf5bff4bf0bb19c90fca9015c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.579663 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 7f416cdb596a4ed5a302acd07bb84b9c (DEFAULT_TABLE table=default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.581065 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21080ab069d74457a9f7fe3aabdb3c7c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.663904 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfd3e263464c4f2c8af9262acf211193" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:38.583209 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b80a91ae2c084020b5d3444b570b28f2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.642736 339 tablet_bootstrap.cc:654] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.664209 1634 raft_consensus.cc:2802] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:38.664243 32514 raft_consensus.cc:2391] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:38.664330 1634 raft_consensus.cc:695] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:38.585404 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.664505 32513 raft_consensus.cc:2391] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:38.664476 32376 raft_consensus.cc:2466] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:38.590214 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d98654c29b9c4e75af6a213cafd4c2c9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.664557 1634 consensus_queue.cc:237] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.591989 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3249605158634f4eab5b9d61357c383f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.664846 32515 raft_consensus.cc:2391] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:38.624279 32483 tablet_service.cc:1468] Processing CreateTablet for tablet a3ad3d261a794d1184061c8e9b384693 (DEFAULT_TABLE table=default.Table_f663b3b4536b4a19a3c42d533d148127 [id=86475d6fed9c4f8b84556bcc1b1030be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.664912 32629 tablet_service.cc:1468] Processing CreateTablet for tablet d4bc93afd147458b833a026e9d0966b8 (DEFAULT_TABLE table=default.Table_f883c19d25584eb185a1e31ef6d88d4f [id=ae1d0fae75be48768a6503498a33df32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.666572 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4bc93afd147458b833a026e9d0966b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.667141 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27da2a90c1b43deb3dc90834f4e312e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.667367 32481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c529e826fa84a919c018887620e4023. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.638669 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:38.667595 32517 raft_consensus.cc:2391] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:38.667695 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f416cdb596a4ed5a302acd07bb84b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.667860 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1494f83d13cd49c7a804593f45f97cc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.668051 32482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9a0c57133884896b03e9c0a8afb8bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.668217 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41b15c89a744a698ca5eee2f143b4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.668363 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bfad68d1f20486997ea1b0a8d97206b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.670892 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aface2a39d764a6cb6e4965867ab7b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.671170 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b7416ad6a244d978457e529cbc789de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.671351 32491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8f8a132e6245f9a04b5bb6f2253c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.671526 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ae0a411a3e54c45b8d5c5b9b0eea762. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.674558 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cee3de5953244f98428747bea912f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.679282 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e01b430a064740b76e0bceec4f5546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.683938 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb99da3b8fa4dc6aa7f11a829d3f2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.689776 32371 raft_consensus.cc:3058] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.691727 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c33fa0a7ff34d2985c5e07fcf88eaa1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.692314 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ad3d261a794d1184061c8e9b384693. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.692380 1659 raft_consensus.cc:491] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7351aa31800245659edc9837e27daffd)
I20250623 08:04:38.692523 1659 raft_consensus.cc:513] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:38.693143 1659 leader_election.cc:290] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:38.699540 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0b2af698b6e4ed0ae66a9bb32be7b31" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:38.701573 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:38.701822 32379 raft_consensus.cc:2391] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:38.702322 32376 raft_consensus.cc:2466] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:38.703239 32627 tablet_service.cc:1468] Processing CreateTablet for tablet ec4d50d5f9394afe97e3b43d65e6894e (DEFAULT_TABLE table=default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.704155 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 8145567741f34d9aa70c5b7b86f7e1c6 (DEFAULT_TABLE table=default.Table_412c3eb097714d94bf0b6c487f1c3caa [id=bbf01747113a4685a0cd746091a9e5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.705345 32636 tablet_service.cc:1468] Processing CreateTablet for tablet e9604d4890c940d8992eaf30993bf622 (DEFAULT_TABLE table=default.Table_dd2b39480694444ba170c65257a8dc58 [id=9dd60e056f8b4c17b9d8d9569ebd3447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.706845 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 4836d621112e46c380c7bc2b7a93117f (DEFAULT_TABLE table=default.Table_afe210848a6b49b2a15d04cf79e1d47c [id=64522ca998ae445696a94c41ca58069a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.707921 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 3054f406c1c34f5d92edeae1b9243777 (DEFAULT_TABLE table=default.Table_a935cce17e794f328640dde460d22086 [id=5847481518254b318e71e8198962267a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.708992 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 3d7773d6479742f79c38eca34018f94f (DEFAULT_TABLE table=default.Table_25f75684255a4b99a8aff38ee935aac5 [id=e85cbbd0d9db4945930b3fca419e3f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.709398 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7773d6479742f79c38eca34018f94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.709467 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4d50d5f9394afe97e3b43d65e6894e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.709651 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4836d621112e46c380c7bc2b7a93117f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.711836 1634 raft_consensus.cc:491] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.711948 1634 raft_consensus.cc:513] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.712494 1634 leader_election.cc:290] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:38.713631 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6926c49b787941d98c647543bd3b9ed5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:38.715121 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9604d4890c940d8992eaf30993bf622. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:38.717397 32589 leader_election.cc:343] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.718892 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8145567741f34d9aa70c5b7b86f7e1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.722985 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3054f406c1c34f5d92edeae1b9243777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.729426 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:38.732075 343 tablet_bootstrap.cc:492] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.732300 343 ts_tablet_manager.cc:1397] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.000s
I20250623 08:04:38.733211 343 raft_consensus.cc:357] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.733484 343 raft_consensus.cc:383] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.733569 343 raft_consensus.cc:738] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.734181 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 178) took 3928 ms. Trace:
I20250623 08:04:38.734303 981 rpcz_store.cc:276] 0623 08:04:34.805517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.805563 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:38.598605 (+3793042us) ts_tablet_manager.cc:1145] Deleting tablet 965f1f0a107b4653b6aac01e0d8e259a
0623 08:04:38.598616 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.607207 (+ 8591us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.614068 (+ 6861us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.734112 (+120044us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3792805}
I20250623 08:04:38.734720 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 179) took 3928 ms. Trace:
I20250623 08:04:38.734843 981 rpcz_store.cc:276] 0623 08:04:34.806010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.806056 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:38.734551 (+3928495us) ts_tablet_manager.cc:1145] Deleting tablet 0311ce67989f49e8866234a88e5b4d09
0623 08:04:38.734561 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.734657 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3928338}
I20250623 08:04:38.741770 342 tablet_bootstrap.cc:492] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:38.741976 342 ts_tablet_manager.cc:1397] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.311s user 0.002s sys 0.002s
I20250623 08:04:38.742643 342 raft_consensus.cc:357] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.743286 32342 tablet_service.cc:1468] Processing CreateTablet for tablet ec4d50d5f9394afe97e3b43d65e6894e (DEFAULT_TABLE table=default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.743503 32355 tablet_service.cc:1468] Processing CreateTablet for tablet e9604d4890c940d8992eaf30993bf622 (DEFAULT_TABLE table=default.Table_dd2b39480694444ba170c65257a8dc58 [id=9dd60e056f8b4c17b9d8d9569ebd3447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.743705 32342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4d50d5f9394afe97e3b43d65e6894e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.743851 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9604d4890c940d8992eaf30993bf622. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.744164 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:38.744383 32649 raft_consensus.cc:2391] T 5f1fb8231e8e45208ac4f412134d8696 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:38.746850 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 4836d621112e46c380c7bc2b7a93117f (DEFAULT_TABLE table=default.Table_afe210848a6b49b2a15d04cf79e1d47c [id=64522ca998ae445696a94c41ca58069a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.746937 32357 tablet_service.cc:1468] Processing CreateTablet for tablet d4bc93afd147458b833a026e9d0966b8 (DEFAULT_TABLE table=default.Table_f883c19d25584eb185a1e31ef6d88d4f [id=ae1d0fae75be48768a6503498a33df32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.747238 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4836d621112e46c380c7bc2b7a93117f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.747284 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 8145567741f34d9aa70c5b7b86f7e1c6 (DEFAULT_TABLE table=default.Table_412c3eb097714d94bf0b6c487f1c3caa [id=bbf01747113a4685a0cd746091a9e5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.747828 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 3054f406c1c34f5d92edeae1b9243777 (DEFAULT_TABLE table=default.Table_a935cce17e794f328640dde460d22086 [id=5847481518254b318e71e8198962267a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:38.748278 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8145567741f34d9aa70c5b7b86f7e1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.750934 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3054f406c1c34f5d92edeae1b9243777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.751292 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f1fb8231e8e45208ac4f412134d8696" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:38.751430 32515 raft_consensus.cc:1273] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Refusing update from remote peer 7351aa31800245659edc9837e27daffd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250623 08:04:38.751495 32517 raft_consensus.cc:2391] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:38.751513 32515 pending_rounds.cc:85] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576: Aborting all ops after (but not including) 0
I20250623 08:04:38.751561 32515 pending_rounds.cc:107] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250623 08:04:38.751652 32515 raft_consensus.cc:2887] T f7d69bcad8684d99a0b08eefb470ab01 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250623 08:04:38.754187 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4bc93afd147458b833a026e9d0966b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:38.733803 343 consensus_queue.cc:260] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.627943 1627 raft_consensus.cc:491] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.755432 343 ts_tablet_manager.cc:1428] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20250623 08:04:38.755501 1627 raft_consensus.cc:513] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.755700 343 tablet_bootstrap.cc:492] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:38.756117 1627 leader_election.cc:290] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.757210 343 tablet_bootstrap.cc:654] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.758351 342 raft_consensus.cc:383] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.758503 342 raft_consensus.cc:738] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.758805 342 consensus_queue.cc:260] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.759486 342 ts_tablet_manager.cc:1428] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250623 08:04:38.759956 342 tablet_bootstrap.cc:492] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:38.761801 342 tablet_bootstrap.cc:654] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.763456 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_bc1251ce708a4467b5b1c71a1df307e0 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 1 got 0
I20250623 08:04:38.763700 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:38.764010 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:38.764102 339 tablet_bootstrap.cc:492] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.764238 339 ts_tablet_manager.cc:1397] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.001s
I20250623 08:04:38.764571 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 90ceb0f8d5c94c4e9427042d8a35e16c (table default.Table_0972bb6b77644e27b56f3e0007bc5fd4 [id=6fa96454486845848e54e86fd64a0017]). Expected version 1 got 0
I20250623 08:04:38.764815 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:38.764955 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]). Expected version 1 got 0
I20250623 08:04:38.764977 339 raft_consensus.cc:357] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.765193 339 raft_consensus.cc:383] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.765261 339 raft_consensus.cc:738] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.765544 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet d5cd316e6f4447be958611ab07ed31fc (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:38.765492 339 consensus_queue.cc:260] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.765750 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:38.766043 339 ts_tablet_manager.cc:1428] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:38.766103 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:38.766363 339 tablet_bootstrap.cc:492] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.766398 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:38.766605 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:38.768266 339 tablet_bootstrap.cc:654] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.772234 339 tablet_bootstrap.cc:492] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.772363 339 ts_tablet_manager.cc:1397] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250623 08:04:38.773063 339 raft_consensus.cc:357] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.773233 339 raft_consensus.cc:383] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.773299 339 raft_consensus.cc:738] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.773521 339 consensus_queue.cc:260] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.773999 339 ts_tablet_manager.cc:1428] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:38.774261 339 tablet_bootstrap.cc:492] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.777814 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 965f1f0a107b4653b6aac01e0d8e259a (table default.Table_5f159a56c60144b99a6847ec6080623e [id=ec1f9e35e20c47fcbe2d4b80c7abeabc]) successfully deleted
I20250623 08:04:38.778549 32314 leader_election.cc:304] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.780156 1622 raft_consensus.cc:3058] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.781868 32314 leader_election.cc:304] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.783115 1303 raft_consensus.cc:3058] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.783283 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:38.783546 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:38.785701 1303 raft_consensus.cc:2747] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.786137 1622 raft_consensus.cc:2747] T 21080ab069d74457a9f7fe3aabdb3c7c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.787067 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:38.786358 32314 leader_election.cc:304] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.787891 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:38.788220 1622 raft_consensus.cc:3058] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.788408 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:38.788908 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:38.789331 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:38.789631 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:38.789861 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:38.790045 32314 leader_election.cc:304] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.791494 1622 raft_consensus.cc:2747] T d98654c29b9c4e75af6a213cafd4c2c9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.791565 1627 raft_consensus.cc:491] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.791632 1303 raft_consensus.cc:3058] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.791702 1627 raft_consensus.cc:513] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.792409 1627 leader_election.cc:290] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.794066 1303 raft_consensus.cc:2747] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.795616 343 tablet_bootstrap.cc:492] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.795864 343 ts_tablet_manager.cc:1397] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.003s
I20250623 08:04:38.797010 343 raft_consensus.cc:357] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.797310 343 raft_consensus.cc:383] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.797437 343 raft_consensus.cc:738] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.797739 343 consensus_queue.cc:260] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.798687 343 ts_tablet_manager.cc:1428] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250623 08:04:38.799075 343 tablet_bootstrap.cc:492] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:38.800457 1622 consensus_queue.cc:1035] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:38.800550 32314 leader_election.cc:304] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.800910 1622 raft_consensus.cc:2747] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.802075 339 tablet_bootstrap.cc:654] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.807101 952 ts_tablet_manager.cc:1918] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.807948 342 tablet_bootstrap.cc:492] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:38.808189 342 ts_tablet_manager.cc:1397] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.000s
I20250623 08:04:38.800875 32314 leader_election.cc:304] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.809315 1622 raft_consensus.cc:2747] T 5f1fb8231e8e45208ac4f412134d8696 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.809221 342 raft_consensus.cc:357] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.809405 342 raft_consensus.cc:383] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.809530 342 raft_consensus.cc:738] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.800604 343 tablet_bootstrap.cc:654] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.810108 342 consensus_queue.cc:260] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.790035 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:38.810829 342 ts_tablet_manager.cc:1428] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:38.811134 342 tablet_bootstrap.cc:492] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:38.812503 342 tablet_bootstrap.cc:654] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.812330 32055 catalog_manager.cc:5582] T f7d69bcad8684d99a0b08eefb470ab01 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 1 to 2, leader changed from 59f7c614254f42009306b13337f06576 (127.31.69.2) to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 2 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:38.813138 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:38.807976 952 log.cc:1199] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/1a070895f4984493a3b9f6fa779797a9
I20250623 08:04:38.813597 952 ts_tablet_manager.cc:1939] T 1a070895f4984493a3b9f6fa779797a9 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:38.813784 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:38.814023 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:38.814082 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 106) took 6355 ms. Trace:
I20250623 08:04:38.814402 952 rpcz_store.cc:276] 0623 08:04:32.459077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.466535 (+ 7458us) service_pool.cc:225] Handling call
0623 08:04:38.632903 (+6166368us) ts_tablet_manager.cc:1145] Deleting tablet 1a070895f4984493a3b9f6fa779797a9
0623 08:04:38.632912 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.752281 (+119369us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.806941 (+ 54660us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.814065 (+ 7124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6166064}
I20250623 08:04:38.814592 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 5 got 0
I20250623 08:04:38.814975 952 tablet_replica.cc:331] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:38.815198 952 raft_consensus.cc:2241] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.815326 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:38.815549 952 raft_consensus.cc:2270] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.815580 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:38.815766 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:38.816018 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:38.816589 952 ts_tablet_manager.cc:1905] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.821600 979 ts_tablet_manager.cc:1918] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:38.821823 979 log.cc:1199] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/20e7a78cfc384033810603316b9e66bf
I20250623 08:04:38.822291 979 ts_tablet_manager.cc:1939] T 20e7a78cfc384033810603316b9e66bf P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.823367 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 258) took 1916 ms. Trace:
I20250623 08:04:38.823488 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 724d662bcf754c74a8f3dff91ee258ae
I20250623 08:04:38.824290 32057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_4d6c00c8083e4821aa4a209e9cee8a91" } new_table_name: "default.Table_133d7b35a6ac4c6b8093e39cd1db48db" modify_external_catalogs: true
I20250623 08:04:38.825476 32052 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet c74e8c697d244ac2ac8174bb1fc76e61
I20250623 08:04:38.826191 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_b68f23253200457683bb5cedcc56041f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:38.826730 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b68f23253200457683bb5cedcc56041f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:38.824757 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 724d662bcf754c74a8f3dff91ee258ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:38 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:38.826265 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet c74e8c697d244ac2ac8174bb1fc76e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:38.826399 343 tablet_bootstrap.cc:492] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.827037 979 rpcz_store.cc:276] 0623 08:04:36.906993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.907047 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:38.553195 (+1646148us) ts_tablet_manager.cc:1145] Deleting tablet 20e7a78cfc384033810603316b9e66bf
0623 08:04:38.553206 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.628401 (+ 75195us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.821483 (+193082us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.823342 (+ 1859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1645993}
I20250623 08:04:38.827234 343 ts_tablet_manager.cc:1397] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20250623 08:04:38.827538 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 20e7a78cfc384033810603316b9e66bf (table default.Table_7987e1a82b644bd69724bc61cd84445a [id=96f6aedcc9704226b4b14e5548a35ad1]) successfully deleted
W20250623 08:04:38.827847 32059 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ecc73f52f0bc43c5bbdca2b89deecb21 consensus state has no leader
I20250623 08:04:38.828186 32059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_79fce459ec304a14823e0a2179f6b804"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:38.828148 343 raft_consensus.cc:357] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.828343 343 raft_consensus.cc:383] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.828416 343 raft_consensus.cc:738] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:38.828620 32059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79fce459ec304a14823e0a2179f6b804 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:38.828676 343 consensus_queue.cc:260] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.828943 979 tablet_replica.cc:331] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.829216 343 ts_tablet_manager.cc:1428] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:38.829568 343 tablet_bootstrap.cc:492] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:38.830071 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_7a30e48550d443ccbe36157dd0e35bcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:38.830521 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a30e48550d443ccbe36157dd0e35bcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:38.831568 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 59015204545e436cb558fe4465a67c97
I20250623 08:04:38.832113 32054 master_service.cc:948] ReplaceTablet: received request to replace tablet bfd3e263464c4f2c8af9262acf211193 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:38.832345 32058 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_1f98d716ca3541379adbced08214bfdd [id=d1fbcd8e89164d71b021f4b1f4223a59] alter complete (version 1)
I20250623 08:04:38.834128 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 59015204545e436cb558fe4465a67c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:38 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:38.846992 343 tablet_bootstrap.cc:654] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.849718 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42c861e29e1e4e549021dae7255a856c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.850302 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36c8714cfbd84dd38cf5c0f305ef6843" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.851531 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa398354a9e44633857a75c0bbba08b3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:38.851701 32517 raft_consensus.cc:2466] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:38.852272 32342 tablet_service.cc:1515] Processing DeleteTablet for tablet 724d662bcf754c74a8f3dff91ee258ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:38 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.852483 32346 tablet_service.cc:1515] Processing DeleteTablet for tablet c74e8c697d244ac2ac8174bb1fc76e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.852502 979 raft_consensus.cc:2241] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.852846 979 raft_consensus.cc:2270] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.853343 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 59015204545e436cb558fe4465a67c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:38 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.854158 979 ts_tablet_manager.cc:1905] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.854609 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa398354a9e44633857a75c0bbba08b3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:38.854806 32517 raft_consensus.cc:3058] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.858400 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_973ac8c9b50b4fb2a8d83318384a8b76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:38.859035 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_973ac8c9b50b4fb2a8d83318384a8b76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:38.860656 32517 raft_consensus.cc:2466] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:38.861313 1682 raft_consensus.cc:491] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.861449 1682 raft_consensus.cc:513] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:38.862133 1682 leader_election.cc:290] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:38.863255 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a0bbafe1ea34ab5ae1b3de25b50d4a3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:38.867280 32314 leader_election.cc:343] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 9a0bbafe1ea34ab5ae1b3de25b50d4a3
I20250623 08:04:38.867483 32314 leader_election.cc:304] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.867987 1682 raft_consensus.cc:2747] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:38.882887 339 tablet_bootstrap.cc:492] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:38.883136 339 ts_tablet_manager.cc:1397] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.109s user 0.005s sys 0.002s
I20250623 08:04:38.883994 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6926c49b787941d98c647543bd3b9ed5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:38.885610 32453 leader_election.cc:343] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:38.887064 32057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b97f3019b893441bae18146fef094bd3 consensus state has no leader
I20250623 08:04:38.890736 339 raft_consensus.cc:357] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.891029 339 raft_consensus.cc:383] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.891146 339 raft_consensus.cc:738] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.891412 339 consensus_queue.cc:260] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.895044 1507 consensus_queue.cc:1035] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:38.896515 1507 consensus_queue.cc:1035] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:38.897125 32453 leader_election.cc:343] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:38.897393 32453 leader_election.cc:343] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:38.899655 32453 leader_election.cc:343] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:38.892134 339 ts_tablet_manager.cc:1428] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250623 08:04:38.903825 339 tablet_bootstrap.cc:492] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:38.905486 339 tablet_bootstrap.cc:654] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.922715 342 tablet_bootstrap.cc:492] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:38.930465 342 ts_tablet_manager.cc:1397] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.001s
I20250623 08:04:38.931748 342 raft_consensus.cc:357] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.932050 342 raft_consensus.cc:383] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.932154 342 raft_consensus.cc:738] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.932492 342 consensus_queue.cc:260] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.933136 342 ts_tablet_manager.cc:1428] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:38.933346 979 ts_tablet_manager.cc:1918] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:38.933495 342 tablet_bootstrap.cc:492] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:38.934154 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 92005abc61dd40ef9da8124cadc29b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.934844 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 358cb6ac24414bdfbc32155e732f9ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.935614 342 tablet_bootstrap.cc:654] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.937444 979 log.cc:1199] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/136e482995a24c5daec55a701f3091fa
I20250623 08:04:38.942435 343 tablet_bootstrap.cc:492] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.942649 343 ts_tablet_manager.cc:1397] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.010s
I20250623 08:04:38.943485 343 raft_consensus.cc:357] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.943734 343 raft_consensus.cc:383] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.943858 343 raft_consensus.cc:738] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:38.944144 343 consensus_queue.cc:260] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.944732 343 ts_tablet_manager.cc:1428] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:38.945076 343 tablet_bootstrap.cc:492] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:38.947073 979 ts_tablet_manager.cc:1939] T 136e482995a24c5daec55a701f3091fa P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:38.947352 1622 raft_consensus.cc:491] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:38.947507 1622 raft_consensus.cc:513] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.947813 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249605158634f4eab5b9d61357c383f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.948024 1622 leader_election.cc:290] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:38.949419 32314 leader_election.cc:343] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: ecb422550efd40cf82bc92fca5c25eda
I20250623 08:04:38.952626 1353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Bootstrap replayed 1/1 log segments. Stats: ops{read=356 overwritten=0 applied=347 ignored=0} inserts{seen=329 ignored=0} mutations{seen=384 ignored=0} orphaned_commits=0. Pending: 9 replicates
I20250623 08:04:38.953467 1353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Bootstrap complete.
I20250623 08:04:38.954319 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet a25bdc9b7e48467fba3771365e58bcbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.948447 32644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecb422550efd40cf82bc92fca5c25eda" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:38.954725 32644 raft_consensus.cc:2391] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
W20250623 08:04:38.954166 32589 consensus_peers.cc:487] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:38.955448 32314 leader_election.cc:304] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:38.956797 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 259) took 2012 ms. Trace:
I20250623 08:04:38.958083 32453 leader_election.cc:304] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:38.958223 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_334749aabade478fbbc2badb1e4960bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:38.958348 1507 raft_consensus.cc:2802] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:38.958456 1507 raft_consensus.cc:695] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
W20250623 08:04:38.958773 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_334749aabade478fbbc2badb1e4960bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:38.958688 1507 consensus_queue.cc:237] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.961257 952 ts_tablet_manager.cc:1918] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:38.961367 952 log.cc:1199] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/3249605158634f4eab5b9d61357c383f
I20250623 08:04:38.961678 952 ts_tablet_manager.cc:1939] T 3249605158634f4eab5b9d61357c383f P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:38.962417 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 111) took 6478 ms. Trace:
I20250623 08:04:38.962495 952 rpcz_store.cc:276] 0623 08:04:32.483890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.483938 (+ 48us) service_pool.cc:225] Handling call
0623 08:04:38.814934 (+6330996us) ts_tablet_manager.cc:1145] Deleting tablet 3249605158634f4eab5b9d61357c383f
0623 08:04:38.814943 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.820306 (+ 5363us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.961148 (+140842us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.962398 (+ 1250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6330795}
I20250623 08:04:38.963050 32453 leader_election.cc:304] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:38.964085 1508 raft_consensus.cc:2802] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:38.964231 1508 raft_consensus.cc:695] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:38.964502 1508 consensus_queue.cc:237] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.967702 1622 raft_consensus.cc:3058] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:38.968828 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.970422 1622 raft_consensus.cc:2747] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:38.970274 979 rpcz_store.cc:276] 0623 08:04:36.944008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.947024 (+ 3016us) service_pool.cc:225] Handling call
0623 08:04:38.828899 (+1881875us) ts_tablet_manager.cc:1145] Deleting tablet 136e482995a24c5daec55a701f3091fa
0623 08:04:38.828908 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.895161 (+ 66253us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.933236 (+ 38075us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.956777 (+ 23541us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23179,"tablet-delete.queue_time_us":1881664}
I20250623 08:04:38.971171 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 136e482995a24c5daec55a701f3091fa (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]) successfully deleted
I20250623 08:04:38.972357 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 485538f7c63948818f20a25c02381519 with delete_type TABLET_DATA_DELETED (replaced by tablet 6857437f85ab40f6a24e74ce3b5efbdc) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.973083 952 tablet_replica.cc:331] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:38.973400 952 raft_consensus.cc:2241] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.973651 952 raft_consensus.cc:2270] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.974329 952 ts_tablet_manager.cc:1905] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.974651 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet effd4fff43fe4d4fb36458986e1dfce1 with delete_type TABLET_DATA_DELETED (replaced by tablet d5cd316e6f4447be958611ab07ed31fc) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:38.975210 979 tablet_replica.cc:331] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:38.980137 979 raft_consensus.cc:2241] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:38.975292 343 tablet_bootstrap.cc:654] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:38.982458 979 pending_rounds.cc:70] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:38.983783 952 ts_tablet_manager.cc:1918] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:38.983986 952 log.cc:1199] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b80a91ae2c084020b5d3444b570b28f2
I20250623 08:04:38.983004 979 pending_rounds.cc:77] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7170727431365124096 op_type: NO_OP noop_request { }
I20250623 08:04:38.984215 979 raft_consensus.cc:2887] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:38.985949 979 raft_consensus.cc:2270] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.989591 952 ts_tablet_manager.cc:1939] T b80a91ae2c084020b5d3444b570b28f2 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:38.990550 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 118) took 6460 ms. Trace:
I20250623 08:04:38.990720 952 rpcz_store.cc:276] 0623 08:04:32.530522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:32.542849 (+ 12327us) service_pool.cc:225] Handling call
0623 08:04:38.973008 (+6430159us) ts_tablet_manager.cc:1145] Deleting tablet b80a91ae2c084020b5d3444b570b28f2
0623 08:04:38.973026 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.978850 (+ 5824us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.983669 (+ 4819us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.990527 (+ 6858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6419997}
I20250623 08:04:38.991300 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 129) took 5853 ms. Trace:
I20250623 08:04:38.992916 979 ts_tablet_manager.cc:1905] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:38.992909 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 0311ce67989f49e8866234a88e5b4d09 because the tablet was not found. No further retry: Not found: Tablet not found: 0311ce67989f49e8866234a88e5b4d09
W20250623 08:04:38.990152 32050 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 89a24019b797467e857b744d701fb1ed consensus state has no leader
I20250623 08:04:38.991432 952 rpcz_store.cc:276] 0623 08:04:33.137521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.137587 (+ 66us) service_pool.cc:225] Handling call
0623 08:04:38.991153 (+5853566us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d1c8a67e149478a0d69842a4eb5b2
0623 08:04:38.991166 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:38.991283 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5853335}
I20250623 08:04:38.996287 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_744bf9c83e004aafb60d2cb44530dff5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:38.996626 952 tablet_replica.cc:331] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:38.996888 952 raft_consensus.cc:2241] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
W20250623 08:04:38.996923 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_744bf9c83e004aafb60d2cb44530dff5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:38.997262 952 raft_consensus.cc:2270] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:38.997835 952 ts_tablet_manager.cc:1905] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:38.998564 32059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_c491a58f8bb24bbf85189ea654bdd45f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:38.998767 343 tablet_bootstrap.cc:492] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:38.998906 343 ts_tablet_manager.cc:1397] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.002s
W20250623 08:04:38.999033 32059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c491a58f8bb24bbf85189ea654bdd45f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:38.999738 343 raft_consensus.cc:357] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:38.999890 343 raft_consensus.cc:383] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:38.999936 343 raft_consensus.cc:738] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.000218 339 tablet_bootstrap.cc:492] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:39.000357 339 ts_tablet_manager.cc:1397] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.000s
I20250623 08:04:39.001051 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet ea702abb31fb41498ab61bbb05b1f7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.001040 339 raft_consensus.cc:357] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.001206 339 raft_consensus.cc:383] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.001267 339 raft_consensus.cc:738] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.001458 339 consensus_queue.cc:260] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.002223 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cb66c7a273a412db3ae77af9e30d036 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.004413 339 ts_tablet_manager.cc:1428] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:39.004753 339 tablet_bootstrap.cc:492] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:39.006834 339 tablet_bootstrap.cc:654] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.008296 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet bfd3e263464c4f2c8af9262acf211193
I20250623 08:04:39.008795 32054 catalog_manager.cc:6652] ReplaceTablet: tablet bfd3e263464c4f2c8af9262acf211193 deleted and replaced by tablet 7f890ee0747b4df0b82bc3b396191124
I20250623 08:04:39.009120 32054 master_service.cc:948] ReplaceTablet: received request to replace tablet a8945b6f49474648ac73bcb85285680a from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:39.009148 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet bfd3e263464c4f2c8af9262acf211193 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f890ee0747b4df0b82bc3b396191124) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.011826 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet c7fcf583064f4538913d9157e93068dd (table default.Table_a65e08da5dbe4b408014e6c3b662450f [id=abda89a30971466ea9a1f410d424ef58]) successfully deleted
I20250623 08:04:39.012828 32589 leader_election.cc:304] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:39.013265 32589 leader_election.cc:343] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.013361 32589 leader_election.cc:304] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.013676 1673 raft_consensus.cc:2802] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:39.013756 1673 raft_consensus.cc:491] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.013818 1673 raft_consensus.cc:3058] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.014209 1506 raft_consensus.cc:2747] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.016559 1673 raft_consensus.cc:513] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.017066 1673 leader_election.cc:290] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.017416 32640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861e45cf546487d9812caa06e2704eb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.017469 952 ts_tablet_manager.cc:1918] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.017598 32640 raft_consensus.cc:2391] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:39.017601 952 log.cc:1199] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f279b63b55c64e548ded27b02025c9f5
I20250623 08:04:39.018018 952 ts_tablet_manager.cc:1939] T f279b63b55c64e548ded27b02025c9f5 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:39.019194 343 consensus_queue.cc:260] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.024869 343 ts_tablet_manager.cc:1428] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250623 08:04:39.025735 32589 leader_election.cc:304] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.026087 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 133) took 5607 ms. Trace:
I20250623 08:04:39.026289 1506 raft_consensus.cc:2747] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.035212 952 rpcz_store.cc:276] 0623 08:04:33.418912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.418979 (+ 67us) service_pool.cc:225] Handling call
0623 08:04:38.996581 (+5577602us) ts_tablet_manager.cc:1145] Deleting tablet f279b63b55c64e548ded27b02025c9f5
0623 08:04:38.996591 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.001028 (+ 4437us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.017326 (+ 16298us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.026065 (+ 8739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5577254}
I20250623 08:04:39.035683 952 tablet_replica.cc:331] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:39.035921 952 raft_consensus.cc:2241] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:39.036155 952 raft_consensus.cc:2270] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.036764 952 ts_tablet_manager.cc:1905] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.038149 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 1a070895f4984493a3b9f6fa779797a9 (table default.Table_79f69fca0b25496e896d93fdd78e14b6 [id=cf095cc479d741b8b7c2cd352724713a]) successfully deleted
W20250623 08:04:39.039182 32589 leader_election.cc:343] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.039287 32589 leader_election.cc:304] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:39.039608 32589 leader_election.cc:343] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.039680 32589 leader_election.cc:304] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.040086 1673 raft_consensus.cc:2747] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:39.040184 32589 leader_election.cc:343] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.040252 1673 raft_consensus.cc:2747] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.042867 343 tablet_bootstrap.cc:492] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:39.044562 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 3249605158634f4eab5b9d61357c383f (table default.Table_79be3d7247284828a82f02d96faa60cc [id=c618740b8cb040bcb78c452b3d747391]) successfully deleted
I20250623 08:04:39.045132 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b80a91ae2c084020b5d3444b570b28f2 (table default.Table_d226f9dab24f45ae896579983b3d1a67 [id=e0cd5a7f63b443c2a3e86ec75a1e3778]) successfully deleted
I20250623 08:04:39.045315 1673 raft_consensus.cc:491] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.045473 1673 raft_consensus.cc:513] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.046036 1673 leader_election.cc:290] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.040263 32589 leader_election.cc:304] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.046789 1673 raft_consensus.cc:2747] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.050228 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_b08756509dd14eeeb598ef00e68600e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:39.050828 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b08756509dd14eeeb598ef00e68600e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.052493 1708 tablet.cc:1722] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:39.055639 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_4903e809cc2d447fb94224ccbb05078d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:39.056207 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4903e809cc2d447fb94224ccbb05078d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.060438 952 ts_tablet_manager.cc:1918] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.060617 952 log.cc:1199] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/358cb6ac24414bdfbc32155e732f9ac8
W20250623 08:04:39.060994 32050 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.1:45613 (request call id 156) took 2005 ms (client timeout 1000 ms). Trace:
I20250623 08:04:39.061415 952 ts_tablet_manager.cc:1939] T 358cb6ac24414bdfbc32155e732f9ac8 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:39.061972 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 134) took 5642 ms. Trace:
I20250623 08:04:39.062168 952 rpcz_store.cc:276] 0623 08:04:33.419535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.419586 (+ 51us) service_pool.cc:225] Handling call
0623 08:04:39.035624 (+5616038us) ts_tablet_manager.cc:1145] Deleting tablet 358cb6ac24414bdfbc32155e732f9ac8
0623 08:04:39.035645 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.044427 (+ 8782us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.060334 (+ 15907us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.061951 (+ 1617us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5615818}
W20250623 08:04:39.061744 32050 rpcz_store.cc:269] 0623 08:04:37.055835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.055894 (+ 59us) service_pool.cc:225] Handling call
0623 08:04:39.060774 (+2004880us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54444,"reported_tablets":176}
I20250623 08:04:39.063520 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 135) took 5589 ms. Trace:
I20250623 08:04:39.063714 952 rpcz_store.cc:276] 0623 08:04:33.473041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.473079 (+ 38us) service_pool.cc:225] Handling call
0623 08:04:39.062549 (+5589470us) ts_tablet_manager.cc:1145] Deleting tablet 9049e3d9b39f420bb51221808409fc54
0623 08:04:39.062558 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.062683 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5586392}
I20250623 08:04:39.064160 952 tablet_replica.cc:331] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:39.008280 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ecf4af9e476495683bd63cac20d4672 with delete_type TABLET_DATA_DELETED (replaced by tablet 52d0a356c74f4daea9093c998b4bf73e) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.012624 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet bfd3e263464c4f2c8af9262acf211193 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f890ee0747b4df0b82bc3b396191124) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.012774 32346 tablet_service.cc:1515] Processing DeleteTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.014621 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc0dad440b494018bc0d6cfd40940bc1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.015007 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861e45cf546487d9812caa06e2704eb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.016697 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 181) took 4209 ms. Trace:
I20250623 08:04:39.020833 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 38ecd4c243b840f4b66d8e01078dca64 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.023166 32342 tablet_service.cc:1515] Processing DeleteTablet for tablet f3d32798168345edb9898751d083057c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.023914 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.025678 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet f279b63b55c64e548ded27b02025c9f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.026943 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861e45cf546487d9812caa06e2704eb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:39.066545 32589 leader_election.cc:343] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.066141 981 rpcz_store.cc:276] 0623 08:04:34.806899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.806937 (+ 38us) service_pool.cc:225] Handling call
0623 08:04:39.016600 (+4209663us) ts_tablet_manager.cc:1145] Deleting tablet 4ac9f38877b24d619c280a1aaa0b7b08
0623 08:04:39.016609 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.016680 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4209477}
I20250623 08:04:39.028391 1696 tablet.cc:1722] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:39.033021 32349 tablet_service.cc:1515] Processing DeleteTablet for tablet b80a91ae2c084020b5d3444b570b28f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.033151 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet dc0dad440b494018bc0d6cfd40940bc1 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f83a6d453ec4ffe85282b4385df5f11) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.033241 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a070895f4984493a3b9f6fa779797a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.037779 32374 raft_consensus.cc:3058] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:39.067116 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 4ac9f38877b24d619c280a1aaa0b7b08 because the tablet was not found. No further retry: Not found: Tablet not found: 4ac9f38877b24d619c280a1aaa0b7b08
I20250623 08:04:39.067445 981 tablet_replica.cc:331] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:39.067637 981 raft_consensus.cc:2241] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.067791 981 raft_consensus.cc:2270] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.068327 981 ts_tablet_manager.cc:1905] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.037927 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa398354a9e44633857a75c0bbba08b3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.070432 32373 raft_consensus.cc:2391] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:39.038818 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 9049e3d9b39f420bb51221808409fc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.038863 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e9bf7cba2244703b7402cb4be1a005e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.040859 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 92005abc61dd40ef9da8124cadc29b6a) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.047664 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b405f87bb1e4d0686a1f627835d901f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.053821 1682 raft_consensus.cc:491] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.071713 1682 raft_consensus.cc:513] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.071928 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_3cbae8872bcd4af3a1a837940945fab6" } new_table_name: "default.Table_217651efa5dc4571b2a47cd3196c9e57" modify_external_catalogs: true
I20250623 08:04:39.072399 1682 leader_election.cc:290] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.073737 32059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_1a60513fe9e94660aa640b5522394bbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:39.072947 32050 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_90920cfec34b4621bae5567c93e73c89" } modify_external_catalogs: true
W20250623 08:04:39.074414 32059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a60513fe9e94660aa640b5522394bbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.071473 1698 tablet.cc:1722] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:39.071580 32589 leader_election.cc:343] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.074527 32050 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_90920cfec34b4621bae5567c93e73c89" } modify_external_catalogs: true
I20250623 08:04:39.075506 979 ts_tablet_manager.cc:1918] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250623 08:04:39.076174 952 raft_consensus.cc:2241] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.076335 952 raft_consensus.cc:2270] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.076437 979 log.cc:1199] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/283bebc7063d437ba3e541d7dee3c0b5
I20250623 08:04:39.076893 952 ts_tablet_manager.cc:1905] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.076973 32652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa398354a9e44633857a75c0bbba08b3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.078563 32639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e9bf7cba2244703b7402cb4be1a005e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.078737 32639 raft_consensus.cc:2466] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:39.079175 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fd478c007914d2cb09d929b50322a64" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.079346 32646 raft_consensus.cc:2391] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:39.075314 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fd478c007914d2cb09d929b50322a64" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:39.081983 32314 leader_election.cc:304] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:39.082288 1682 raft_consensus.cc:3058] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.084985 979 ts_tablet_manager.cc:1939] T 283bebc7063d437ba3e541d7dee3c0b5 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:39.085669 1682 raft_consensus.cc:2747] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.086380 32353 tablet_service.cc:1468] Processing CreateTablet for tablet a375f8ea383b44a9acd04fc2171d34ae (DEFAULT_TABLE table=default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.086786 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a375f8ea383b44a9acd04fc2171d34ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.086704 32627 tablet_service.cc:1468] Processing CreateTablet for tablet a375f8ea383b44a9acd04fc2171d34ae (DEFAULT_TABLE table=default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.087069 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a375f8ea383b44a9acd04fc2171d34ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.090703 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 260) took 2110 ms. Trace:
I20250623 08:04:39.091295 979 rpcz_store.cc:276] 0623 08:04:36.980203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.980266 (+ 63us) service_pool.cc:225] Handling call
0623 08:04:38.973914 (+1993648us) ts_tablet_manager.cc:1145] Deleting tablet 283bebc7063d437ba3e541d7dee3c0b5
0623 08:04:38.974114 (+ 200us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.033528 (+ 59414us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.074719 (+ 41191us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.090680 (+ 15961us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1993211}
I20250623 08:04:39.092613 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 283bebc7063d437ba3e541d7dee3c0b5 (table default.Table_c390976ce10f40d193a81eae44cd862f [id=c8fe0110bcd04cc2a14886d4f3a81dec]) successfully deleted
I20250623 08:04:39.092957 32053 master_service.cc:948] ReplaceTablet: received request to replace tablet f0e24d8d890d4d1db051f172b3ec96ec from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:39.093942 979 tablet_replica.cc:331] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:39.095028 979 raft_consensus.cc:2241] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.096005 979 raft_consensus.cc:2270] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:39.097661 32453 leader_election.cc:343] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 18ea63263b9b466e922169eecafab747
I20250623 08:04:39.097791 32453 leader_election.cc:304] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.098030 1508 raft_consensus.cc:2747] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.098371 32453 leader_election.cc:304] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.098591 1359 raft_consensus.cc:2802] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:39.098671 1359 raft_consensus.cc:491] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.098848 1359 raft_consensus.cc:3058] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.099962 979 ts_tablet_manager.cc:1905] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.094054 1353 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 2 FOLLOWER]: Replica starting. Triggering 9 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:39.101930 1359 raft_consensus.cc:513] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.102511 1359 leader_election.cc:290] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:39.107259 32057 master_service.cc:948] ReplaceTablet: received request to replace tablet f4a96ae9190640dea11bd11a9f0e4372 from {username='slave'} at 127.0.0.1:36116
W20250623 08:04:39.109158 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2c990297f554adbb2d97f6309758940 consensus state has no leader
I20250623 08:04:39.109539 32051 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_75451c4863a34537a328617c53d46cae" } modify_external_catalogs: true
I20250623 08:04:39.109685 952 ts_tablet_manager.cc:1918] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:39.109715 32051 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_75451c4863a34537a328617c53d46cae" } modify_external_catalogs: true
I20250623 08:04:39.109784 952 log.cc:1199] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/dc0dad440b494018bc0d6cfd40940bc1
I20250623 08:04:39.110086 952 ts_tablet_manager.cc:1939] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:39.110489 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 137) took 5583 ms. Trace:
I20250623 08:04:39.110561 952 rpcz_store.cc:276] 0623 08:04:33.527274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.527413 (+ 139us) service_pool.cc:225] Handling call
0623 08:04:39.064105 (+5536692us) ts_tablet_manager.cc:1145] Deleting tablet dc0dad440b494018bc0d6cfd40940bc1
0623 08:04:39.064117 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.105009 (+ 40892us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.109592 (+ 4583us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.110469 (+ 877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5536421}
I20250623 08:04:39.110903 952 tablet_replica.cc:331] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:39.114504 952 raft_consensus.cc:2241] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.121628 343 tablet_bootstrap.cc:654] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.123914 32059 master_service.cc:948] ReplaceTablet: received request to replace tablet 803d3607d75443568b7ddc9f0c0bdc90 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:39.126061 339 tablet_bootstrap.cc:492] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:39.126202 339 ts_tablet_manager.cc:1397] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.122s user 0.002s sys 0.001s
I20250623 08:04:39.127175 339 raft_consensus.cc:357] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.127568 339 raft_consensus.cc:383] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.127686 339 raft_consensus.cc:738] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.127966 339 consensus_queue.cc:260] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.128675 339 ts_tablet_manager.cc:1428] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:39.128990 339 tablet_bootstrap.cc:492] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:39.129009 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f0e24d8d890d4d1db051f172b3ec96ec
I20250623 08:04:39.129596 32053 catalog_manager.cc:6652] ReplaceTablet: tablet f0e24d8d890d4d1db051f172b3ec96ec deleted and replaced by tablet a1cf375214c54f3491098db07faedd9f
I20250623 08:04:39.129827 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet a8945b6f49474648ac73bcb85285680a
I20250623 08:04:39.130182 32051 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f02f36d6c1174a23a1f1043a431d0b6b
I20250623 08:04:39.130239 32054 catalog_manager.cc:6652] ReplaceTablet: tablet a8945b6f49474648ac73bcb85285680a deleted and replaced by tablet 1a404aac062c41e8a06207d9d8685a4d
I20250623 08:04:39.130792 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet f4a96ae9190640dea11bd11a9f0e4372
I20250623 08:04:39.130874 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet a8945b6f49474648ac73bcb85285680a with delete_type TABLET_DATA_DELETED (replaced by tablet 1a404aac062c41e8a06207d9d8685a4d) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.130972 339 tablet_bootstrap.cc:654] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.131181 32057 catalog_manager.cc:6652] ReplaceTablet: tablet f4a96ae9190640dea11bd11a9f0e4372 deleted and replaced by tablet 5b1a601763e44934a7db2f53d05e43a5
I20250623 08:04:39.131577 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet a8945b6f49474648ac73bcb85285680a with delete_type TABLET_DATA_DELETED (replaced by tablet 1a404aac062c41e8a06207d9d8685a4d) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.131799 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet f02f36d6c1174a23a1f1043a431d0b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.131955 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a96ae9190640dea11bd11a9f0e4372 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b1a601763e44934a7db2f53d05e43a5) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.132076 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet f02f36d6c1174a23a1f1043a431d0b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.132076 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a96ae9190640dea11bd11a9f0e4372 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b1a601763e44934a7db2f53d05e43a5) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.132288 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet f0e24d8d890d4d1db051f172b3ec96ec with delete_type TABLET_DATA_DELETED (replaced by tablet a1cf375214c54f3491098db07faedd9f) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.132511 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet f0e24d8d890d4d1db051f172b3ec96ec with delete_type TABLET_DATA_DELETED (replaced by tablet a1cf375214c54f3491098db07faedd9f) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.135610 1707 raft_consensus.cc:491] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.135754 1707 raft_consensus.cc:513] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.136492 1707 leader_election.cc:290] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.137328 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7874a1d5cb9a464294ca32b2767efbf3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:39.138165 32589 leader_election.cc:343] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.139940 952 raft_consensus.cc:2270] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.140738 952 ts_tablet_manager.cc:1905] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.141235 981 ts_tablet_manager.cc:1918] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.141463 981 log.cc:1199] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/f4a0b9402a264800a1ea39707c10da0c
I20250623 08:04:39.142140 981 ts_tablet_manager.cc:1939] T f4a0b9402a264800a1ea39707c10da0c P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:39.111271 32453 leader_election.cc:304] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.142776 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 182) took 4335 ms. Trace:
I20250623 08:04:39.143028 1359 raft_consensus.cc:2802] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:39.143405 1508 consensus_queue.cc:1035] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:39.143577 32453 leader_election.cc:304] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.143817 1508 raft_consensus.cc:2802] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:39.143922 1508 raft_consensus.cc:695] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:39.143986 342 tablet_bootstrap.cc:492] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:39.144104 342 ts_tablet_manager.cc:1397] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.211s user 0.010s sys 0.021s
I20250623 08:04:39.144147 1508 consensus_queue.cc:237] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.144824 342 raft_consensus.cc:357] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.144956 342 raft_consensus.cc:383] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.144999 342 raft_consensus.cc:738] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.145145 342 consensus_queue.cc:260] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.145571 342 ts_tablet_manager.cc:1428] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250623 08:04:39.147044 1707 raft_consensus.cc:491] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.147183 1707 raft_consensus.cc:513] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.147811 1707 leader_election.cc:290] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.151643 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73453eb1a0394cfe8d49adcc71d4f3e9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:39.152196 32589 leader_election.cc:343] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.142948 981 rpcz_store.cc:276] 0623 08:04:34.807285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.807329 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:39.067387 (+4260058us) ts_tablet_manager.cc:1145] Deleting tablet f4a0b9402a264800a1ea39707c10da0c
0623 08:04:39.067405 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.095547 (+ 28142us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.141032 (+ 45485us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.142709 (+ 1677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4259905}
I20250623 08:04:39.153759 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet f4a0b9402a264800a1ea39707c10da0c (table default.Table_244bcdc8d69b4f61a337dac403ffe6cb [id=b3ab55b66d754642aed2f746d9a35807]) successfully deleted
I20250623 08:04:39.155541 981 tablet_replica.cc:331] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:39.156085 981 raft_consensus.cc:2241] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.156821 952 ts_tablet_manager.cc:1918] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.160154 343 tablet_bootstrap.cc:492] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:39.160385 343 ts_tablet_manager.cc:1397] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.000s
I20250623 08:04:39.161598 343 raft_consensus.cc:357] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.161914 343 raft_consensus.cc:383] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.162026 343 raft_consensus.cc:738] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.162330 343 consensus_queue.cc:260] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.156504 981 raft_consensus.cc:2270] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.164920 981 ts_tablet_manager.cc:1905] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.157562 1359 raft_consensus.cc:695] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:39.165262 952 log.cc:1199] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/485538f7c63948818f20a25c02381519
I20250623 08:04:39.165464 1359 consensus_queue.cc:237] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.166044 952 ts_tablet_manager.cc:1939] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:39.166926 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 136) took 5639 ms. Trace:
I20250623 08:04:39.168352 343 ts_tablet_manager.cc:1428] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250623 08:04:39.168695 343 tablet_bootstrap.cc:492] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:39.169390 952 rpcz_store.cc:276] 0623 08:04:33.527117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.527705 (+ 588us) service_pool.cc:225] Handling call
0623 08:04:39.110846 (+5583141us) ts_tablet_manager.cc:1145] Deleting tablet 485538f7c63948818f20a25c02381519
0623 08:04:39.110857 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.150070 (+ 39213us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.156717 (+ 6647us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.166909 (+ 10192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5582836}
I20250623 08:04:39.170063 952 tablet_replica.cc:331] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:39.172756 1353 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9752155c20ee42fbb7ff7b7d5b682444, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.173632 1353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 347, Last appended: 2.356, Last appended by leader: 356, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:39.174516 952 raft_consensus.cc:2241] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.174856 952 raft_consensus.cc:2270] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.175415 342 tablet_bootstrap.cc:492] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:39.175679 952 ts_tablet_manager.cc:1905] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.178714 342 tablet_bootstrap.cc:654] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.180711 1707 raft_consensus.cc:491] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.180855 1707 raft_consensus.cc:513] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.181510 1707 leader_election.cc:290] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.181573 32053 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0b9112de297847f889ab29b68f4dfb8c" } modify_external_catalogs: true
I20250623 08:04:39.181788 32053 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0b9112de297847f889ab29b68f4dfb8c" } modify_external_catalogs: true
W20250623 08:04:39.182042 32453 consensus_peers.cc:487] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20e7a78cfc384033810603316b9e66bf. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:39.182442 32453 leader_election.cc:343] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: dc0dad440b494018bc0d6cfd40940bc1
W20250623 08:04:39.183118 32453 consensus_peers.cc:487] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4a0b9402a264800a1ea39707c10da0c. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.183311 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_f663b3b4536b4a19a3c42d533d148127" } new_table_name: "default.Table_5eda9b192e4f4d66b647be71493a31a5" modify_external_catalogs: true
I20250623 08:04:39.183460 32453 leader_election.cc:304] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:39.183073 979 ts_tablet_manager.cc:1918] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:39.183717 1659 raft_consensus.cc:2802] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250623 08:04:39.183746 32453 consensus_peers.cc:487] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a414144276f6484ea4690204752e2142. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.183799 1659 raft_consensus.cc:491] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.183864 1659 raft_consensus.cc:3058] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:39.184257 32453 leader_election.cc:343] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 485538f7c63948818f20a25c02381519
I20250623 08:04:39.184123 979 log.cc:1199] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/771ecfc41e2b4007a4d213cec5b128c6
I20250623 08:04:39.184712 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_bcdbc5ed766f43ba91aaa4711f0b24f6" } new_table_name: "default.Table_46ce438e1ca54a23abcbf79c87396f71" modify_external_catalogs: true
I20250623 08:04:39.185215 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 1aa604f500984e3b969e15592cdbed6f (DEFAULT_TABLE table=default.Table_54fe37774d62482296afe3191f2e8e88 [id=20b80e6a4e76480890e6d28402d54dc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.185020 343 tablet_bootstrap.cc:654] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.185587 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa604f500984e3b969e15592cdbed6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.185943 32057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_08ce069e682d42acac14ed36e986d074" } modify_external_catalogs: true
I20250623 08:04:39.186113 32057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_08ce069e682d42acac14ed36e986d074" } modify_external_catalogs: true
I20250623 08:04:39.186620 1659 raft_consensus.cc:513] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.186656 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a96ae9190640dea11bd11a9f0e4372" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.186911 979 ts_tablet_manager.cc:1939] T 771ecfc41e2b4007a4d213cec5b128c6 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:39.187268 32589 leader_election.cc:343] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.187817 1659 leader_election.cc:290] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:39.194453 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 3d7773d6479742f79c38eca34018f94f (DEFAULT_TABLE table=default.Table_25f75684255a4b99a8aff38ee935aac5 [id=e85cbbd0d9db4945930b3fca419e3f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.194619 32492 tablet_service.cc:1468] Processing CreateTablet for tablet ec4d50d5f9394afe97e3b43d65e6894e (DEFAULT_TABLE table=default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.195088 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec4d50d5f9394afe97e3b43d65e6894e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.195724 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet ba3d2fa01d574dd992dc904ac024ba3a
I20250623 08:04:39.196395 32053 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0972bb6b77644e27b56f3e0007bc5fd4" } modify_external_catalogs: true
I20250623 08:04:39.196601 32053 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0972bb6b77644e27b56f3e0007bc5fd4" } modify_external_catalogs: true
W20250623 08:04:39.197134 32054 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3ad3d261a794d1184061c8e9b384693 consensus state has no leader
I20250623 08:04:39.197398 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet ba3d2fa01d574dd992dc904ac024ba3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.197521 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_ba8a245f5a564519b522c175b4117f52" } new_table_name: "default.Table_91d43acff7904e08ac276de95b652a5c" modify_external_catalogs: true
I20250623 08:04:39.201714 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:39.202560 1508 raft_consensus.cc:491] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.202713 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 261) took 2221 ms. Trace:
I20250623 08:04:39.202883 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet ba3d2fa01d574dd992dc904ac024ba3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.202868 979 rpcz_store.cc:276] 0623 08:04:36.981211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.981258 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:39.093745 (+2112487us) ts_tablet_manager.cc:1145] Deleting tablet 771ecfc41e2b4007a4d213cec5b128c6
0623 08:04:39.093754 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.144532 (+ 50778us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.182079 (+ 37547us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.202660 (+ 20581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2111961}
I20250623 08:04:39.203328 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:39.203574 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 771ecfc41e2b4007a4d213cec5b128c6 (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]) successfully deleted
I20250623 08:04:39.203824 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:39.202690 1508 raft_consensus.cc:513] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.204058 1659 raft_consensus.cc:491] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.204531 1659 raft_consensus.cc:513] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.204851 1508 leader_election.cc:290] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:39.205116 1659 leader_election.cc:290] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:39.205986 339 tablet_bootstrap.cc:492] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
W20250623 08:04:39.206164 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b4e01b430a064740b76e0bceec4f5546 consensus state has no leader
I20250623 08:04:39.206168 339 ts_tablet_manager.cc:1397] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.000s
I20250623 08:04:39.206971 339 raft_consensus.cc:357] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.207130 339 raft_consensus.cc:383] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.207178 339 raft_consensus.cc:738] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.207345 339 consensus_queue.cc:260] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.207337 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_bc1251ce708a4467b5b1c71a1df307e0" } new_table_name: "default.Table_483c90d6ac4445e482e0eedd47c52bb8" modify_external_catalogs: true
I20250623 08:04:39.209376 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:39.209774 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:39.209990 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:39.210230 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:39.210438 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:39.210681 32052 catalog_manager.cc:5582] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:39.211110 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 5 got 0
I20250623 08:04:39.211470 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:39.214234 1353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: configured and running, proceeding with master startup.
I20250623 08:04:39.215355 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d7773d6479742f79c38eca34018f94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.211637 32052 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:39.218788 32052 catalog_manager.cc:5582] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:39.219101 32484 tablet_service.cc:1468] Processing CreateTablet for tablet d4bc93afd147458b833a026e9d0966b8 (DEFAULT_TABLE table=default.Table_f883c19d25584eb185a1e31ef6d88d4f [id=ae1d0fae75be48768a6503498a33df32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.219451 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4bc93afd147458b833a026e9d0966b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.220692 1267 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 2 FOLLOWER]: Advancing to term 3
I20250623 08:04:39.221726 32627 tablet_service.cc:1468] Processing CreateTablet for tablet f20b23fd6e5c44f58b1fb2bf617c5704 (DEFAULT_TABLE table=default.Table_744bf9c83e004aafb60d2cb44530dff5 [id=dd185047529940ebbd12e27a9f6577a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.222863 32058 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9] alter complete (version 2)
I20250623 08:04:39.223845 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 with delete_type TABLET_DATA_DELETED (replaced by tablet 306c4021b5d94a4996de07c222a8bfd8) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.224249 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 13824916241c4cf596119d44d89cd17f (DEFAULT_TABLE table=default.Table_c491a58f8bb24bbf85189ea654bdd45f [id=826cd9624f2a4f818f02427a9462a9b2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:39.224447 32058 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:39129 (request call id 175) took 1401 ms (client timeout 1000 ms). Trace:
W20250623 08:04:39.224546 32058 rpcz_store.cc:269] 0623 08:04:37.823287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.826432 (+ 3145us) service_pool.cc:225] Handling call
0623 08:04:37.847861 (+ 21429us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f3b58 00007f19c7640176 00007f19c763fea5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:39.224389 (+1376528us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:37.964277 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:37.964402 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:37.964408 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:37.964410 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:37.964514 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:37.964525 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:37.964530 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:37.964593 (+ 63us) tablet.cc:699] Row locks acquired
0623 08:04:37.964596 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:37.964628 (+ 32us) write_op.cc:270] Start()
0623 08:04:37.964675 (+ 47us) write_op.cc:276] Timestamp: P: 1750665877964597 usec, L: 0
0623 08:04:37.964679 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:37.964826 (+ 147us) log.cc:844] Serialized 356 byte log entry
0623 08:04:37.996934 (+ 32108us) op_driver.cc:464] REPLICATION: finished
0623 08:04:37.997648 (+ 714us) write_op.cc:301] APPLY: starting
0623 08:04:37.997662 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:37.997743 (+ 81us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:37.997827 (+ 84us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:37.997876 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:37.997913 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:37.997935 (+ 22us) write_op.cc:312] APPLY: finished
0623 08:04:37.998006 (+ 71us) log.cc:844] Serialized 23 byte log entry
0623 08:04:37.998052 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:37.998067 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:37.998074 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0623 08:04:38.997508 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:38.997620 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:38.997626 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:38.997628 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:38.997730 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:38.997741 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:38.997745 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:38.997806 (+ 61us) tablet.cc:699] Row locks acquired
0623 08:04:38.997808 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:38.997841 (+ 33us) write_op.cc:270] Start()
0623 08:04:38.997875 (+ 34us) write_op.cc:276] Timestamp: P: 1750665878997813 usec, L: 0
0623 08:04:38.997881 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:38.998070 (+ 189us) log.cc:844] Serialized 356 byte log entry
0623 08:04:39.005531 (+ 7461us) op_driver.cc:464] REPLICATION: finished
0623 08:04:39.007583 (+ 2052us) write_op.cc:301] APPLY: starting
0623 08:04:39.007592 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:39.007620 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:39.007641 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:39.007676 (+ 35us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:39.007700 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:39.007707 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:39.007756 (+ 49us) log.cc:844] Serialized 23 byte log entry
0623 08:04:39.007788 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.007800 (+ 12us) write_op.cc:454] Released schema lock
0623 08:04:39.007804 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:39.043878 (+ 36074us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.043891 (+ 13us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":18,"spinlock_wait_cycles":46948224,"child_traces":[["op",{"apply.queue_time_us":665,"apply.run_cpu_time_us":505,"apply.run_wall_time_us":505,"num_ops":1,"prepare.queue_time_us":3732,"prepare.run_cpu_time_us":602,"prepare.run_wall_time_us":601,"replication_time_us":32243}],["op",{"apply.queue_time_us":2023,"apply.run_cpu_time_us":276,"apply.run_wall_time_us":276,"num_ops":1,"prepare.queue_time_us":119109,"prepare.run_cpu_time_us":6609,"prepare.run_wall_time_us":46348,"raft.queue_time_us":4961,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":7633,"thread_start_us":45653,"threads_started":1}]]}
I20250623 08:04:39.225826 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.225767 32058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_c4c658393f424eb1ac22133856d0d0b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:39.225999 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet a414144276f6484ea4690204752e2142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.226146 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 52d0a356c74f4daea9093c998b4bf73e with delete_type TABLET_DATA_DELETED (replaced by tablet bb06a340f82047ac8ec3516edb08f4b6) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.226284 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb422550efd40cf82bc92fca5c25eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:39.226349 32058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4c658393f424eb1ac22133856d0d0b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.226380 1707 raft_consensus.cc:491] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.226490 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249605158634f4eab5b9d61357c383f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.226531 1707 raft_consensus.cc:513] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.226657 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a0b9402a264800a1ea39707c10da0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.226890 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 136e482995a24c5daec55a701f3091fa with delete_type TABLET_DATA_DELETED (replaced by tablet 3bfad68d1f20486997ea1b0a8d97206b) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.227155 1707 leader_election.cc:290] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.227838 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38eecf4a94d4536b98d4f6e037a0094" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.227177 343 tablet_bootstrap.cc:492] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:39.228281 343 ts_tablet_manager.cc:1397] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.001s
W20250623 08:04:39.228600 32589 leader_election.cc:343] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.229048 343 raft_consensus.cc:357] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.229203 343 raft_consensus.cc:383] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.229255 343 raft_consensus.cc:738] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.229292 32428 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:39.229476 343 consensus_queue.cc:260] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.230530 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 115763354f294954a260b0aa5c66a146 (DEFAULT_TABLE table=default.Table_973ac8c9b50b4fb2a8d83318384a8b76 [id=ab80a8647b644e6ebbdfba0d2e76107a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.230953 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 115763354f294954a260b0aa5c66a146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.231120 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 115763354f294954a260b0aa5c66a146 (DEFAULT_TABLE table=default.Table_973ac8c9b50b4fb2a8d83318384a8b76 [id=ab80a8647b644e6ebbdfba0d2e76107a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.232151 343 ts_tablet_manager.cc:1428] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250623 08:04:39.232599 339 ts_tablet_manager.cc:1428] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250623 08:04:39.232870 339 tablet_bootstrap.cc:492] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:39.233922 981 ts_tablet_manager.cc:1918] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.234018 981 log.cc:1199] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/cdec6e9573b041e0a57d1e0e0239d97f
I20250623 08:04:39.234592 339 tablet_bootstrap.cc:654] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.234997 981 ts_tablet_manager.cc:1939] T cdec6e9573b041e0a57d1e0e0239d97f P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:39.235016 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 13824916241c4cf596119d44d89cd17f (DEFAULT_TABLE table=default.Table_c491a58f8bb24bbf85189ea654bdd45f [id=826cd9624f2a4f818f02427a9462a9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.235359 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13824916241c4cf596119d44d89cd17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.238569 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet e9f8aa424bbb4aa8a907259270d46656 with delete_type TABLET_DATA_DELETED (replaced by tablet a0e9b2f810014e81a29ca9b41cd3f8ba) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.239212 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 0749cb735acf4cb393ad78a2cd0c0d60 (DEFAULT_TABLE table=default.Table_7a30e48550d443ccbe36157dd0e35bcc [id=7a1f259c8fc24b4ca83cfa09e5673d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.239531 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0749cb735acf4cb393ad78a2cd0c0d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.243343 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 0749cb735acf4cb393ad78a2cd0c0d60 (DEFAULT_TABLE table=default.Table_7a30e48550d443ccbe36157dd0e35bcc [id=7a1f259c8fc24b4ca83cfa09e5673d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.245659 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 262) took 2263 ms. Trace:
I20250623 08:04:39.245823 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f20b23fd6e5c44f58b1fb2bf617c5704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.245757 979 rpcz_store.cc:276] 0623 08:04:36.982079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.982123 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:39.221520 (+2239397us) ts_tablet_manager.cc:1145] Deleting tablet a34da529ffde4595a23fc6596b30e3e9
0623 08:04:39.245491 (+ 23971us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.245636 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2239258}
W20250623 08:04:39.246306 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet a34da529ffde4595a23fc6596b30e3e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a34da529ffde4595a23fc6596b30e3e9 already in progress: creating tablet
I20250623 08:04:39.246609 979 tablet_replica.cc:331] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:39.246826 979 raft_consensus.cc:2241] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.246994 979 raft_consensus.cc:2270] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.247059 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13824916241c4cf596119d44d89cd17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.247488 979 ts_tablet_manager.cc:1905] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.251251 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 183) took 4443 ms. Trace:
I20250623 08:04:39.251348 981 rpcz_store.cc:276] 0623 08:04:34.807659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.807701 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:39.155420 (+4347719us) ts_tablet_manager.cc:1145] Deleting tablet cdec6e9573b041e0a57d1e0e0239d97f
0623 08:04:39.155458 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.208241 (+ 52783us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.233822 (+ 25581us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.251228 (+ 17406us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4347531}
I20250623 08:04:39.251637 981 tablet_replica.cc:331] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:39.251806 981 raft_consensus.cc:2241] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.251926 981 raft_consensus.cc:2270] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.256167 343 tablet_bootstrap.cc:492] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:39.257964 343 tablet_bootstrap.cc:654] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.258144 32353 tablet_service.cc:1468] Processing CreateTablet for tablet f20b23fd6e5c44f58b1fb2bf617c5704 (DEFAULT_TABLE table=default.Table_744bf9c83e004aafb60d2cb44530dff5 [id=dd185047529940ebbd12e27a9f6577a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.258510 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f20b23fd6e5c44f58b1fb2bf617c5704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.258548 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet cdec6e9573b041e0a57d1e0e0239d97f (table default.Table_48cbdc58fd034e6390087ed7c6346d24 [id=ef1a8122b8044388b4ae24a03da26837]) successfully deleted
I20250623 08:04:39.259753 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 115763354f294954a260b0aa5c66a146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.259845 979 ts_tablet_manager.cc:1918] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250623 08:04:39.259963 979 log.cc:1199] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/8c01121f2e644d6aa6e8ec228bf75be9
I20250623 08:04:39.260254 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0749cb735acf4cb393ad78a2cd0c0d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.260375 979 ts_tablet_manager.cc:1939] T 8c01121f2e644d6aa6e8ec228bf75be9 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:39.261180 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 263) took 2244 ms. Trace:
I20250623 08:04:39.261296 979 rpcz_store.cc:276] 0623 08:04:37.016921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.016993 (+ 72us) service_pool.cc:225] Handling call
0623 08:04:39.246566 (+2229573us) ts_tablet_manager.cc:1145] Deleting tablet 8c01121f2e644d6aa6e8ec228bf75be9
0623 08:04:39.246577 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.250648 (+ 4071us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.259733 (+ 9085us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.261160 (+ 1427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2229344}
I20250623 08:04:39.261839 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 8c01121f2e644d6aa6e8ec228bf75be9 (table default.Table_3af33890e8ff40c881af74580254633b [id=8b20894a91c54a3eb08d5e1b4a42d269]) successfully deleted
I20250623 08:04:39.270077 1682 raft_consensus.cc:491] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.270272 1682 raft_consensus.cc:513] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.273314 981 ts_tablet_manager.cc:1905] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.274169 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 264) took 1987 ms. Trace:
I20250623 08:04:39.274253 979 rpcz_store.cc:276] 0623 08:04:37.287213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.287270 (+ 57us) service_pool.cc:225] Handling call
0623 08:04:39.270956 (+1983686us) ts_tablet_manager.cc:1145] Deleting tablet 6b04e38d3da44301a450fe1fb5e2ada6
0623 08:04:39.274038 (+ 3082us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.274149 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1983506}
W20250623 08:04:39.274807 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 6b04e38d3da44301a450fe1fb5e2ada6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b04e38d3da44301a450fe1fb5e2ada6 already in progress: creating tablet
I20250623 08:04:39.275779 1682 leader_election.cc:290] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:39.276465 32314 leader_election.cc:343] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: e9f8aa424bbb4aa8a907259270d46656
I20250623 08:04:39.276440 32484 tablet_service.cc:1468] Processing CreateTablet for tablet 8145567741f34d9aa70c5b7b86f7e1c6 (DEFAULT_TABLE table=default.Table_412c3eb097714d94bf0b6c487f1c3caa [id=bbf01747113a4685a0cd746091a9e5bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.276485 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f8aa424bbb4aa8a907259270d46656" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.276732 32655 raft_consensus.cc:2466] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:39.276844 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8145567741f34d9aa70c5b7b86f7e1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.277113 32314 leader_election.cc:304] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:39.277346 1682 raft_consensus.cc:2802] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:39.277428 1682 raft_consensus.cc:491] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.277487 1682 raft_consensus.cc:3058] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.278221 32492 tablet_service.cc:1468] Processing CreateTablet for tablet e9604d4890c940d8992eaf30993bf622 (DEFAULT_TABLE table=default.Table_dd2b39480694444ba170c65257a8dc58 [id=9dd60e056f8b4c17b9d8d9569ebd3447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.278587 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9604d4890c940d8992eaf30993bf622. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.279986 343 tablet_bootstrap.cc:492] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:39.280175 343 ts_tablet_manager.cc:1397] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20250623 08:04:39.280980 343 raft_consensus.cc:357] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.281191 343 raft_consensus.cc:383] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.281284 343 raft_consensus.cc:738] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.281565 343 consensus_queue.cc:260] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.282464 343 ts_tablet_manager.cc:1428] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:39.291891 1682 raft_consensus.cc:513] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.298303 32627 tablet_service.cc:1468] Processing CreateTablet for tablet ca5c694856f44201a8aa4314a4f81681 (DEFAULT_TABLE table=default.Table_b68f23253200457683bb5cedcc56041f [id=4f81957f51bd40c192ad7c716cbd893f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.300488 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 1b213c8c685c43868df1e814c79fc61f
I20250623 08:04:39.302237 32050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159
I20250623 08:04:39.302960 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_fddbef2bc1d046689a8ef37cdcddcfa4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:39.303216 339 tablet_bootstrap.cc:492] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:39.303342 339 ts_tablet_manager.cc:1397] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.003s
W20250623 08:04:39.303583 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fddbef2bc1d046689a8ef37cdcddcfa4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.304010 343 tablet_bootstrap.cc:492] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:39.304118 339 raft_consensus.cc:357] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.304283 339 raft_consensus.cc:383] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.304332 339 raft_consensus.cc:738] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.304543 339 consensus_queue.cc:260] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.305112 1682 leader_election.cc:290] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:39.305368 32054 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f957f5575e9475eb9f639e3ad5cb809 consensus state has no leader
I20250623 08:04:39.305593 32353 tablet_service.cc:1468] Processing CreateTablet for tablet ca5c694856f44201a8aa4314a4f81681 (DEFAULT_TABLE table=default.Table_b68f23253200457683bb5cedcc56041f [id=4f81957f51bd40c192ad7c716cbd893f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.305721 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_1911a70d3e224ae4b9805e345c99624b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:39.305985 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5c694856f44201a8aa4314a4f81681. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:39.306197 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1911a70d3e224ae4b9805e345c99624b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.306473 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 4836d621112e46c380c7bc2b7a93117f (DEFAULT_TABLE table=default.Table_afe210848a6b49b2a15d04cf79e1d47c [id=64522ca998ae445696a94c41ca58069a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.306633 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 3054f406c1c34f5d92edeae1b9243777 (DEFAULT_TABLE table=default.Table_a935cce17e794f328640dde460d22086 [id=5847481518254b318e71e8198962267a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.306871 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4836d621112e46c380c7bc2b7a93117f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.307013 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3054f406c1c34f5d92edeae1b9243777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.308959 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_8e6f4c63178440f4ba5c9f65d213df3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:39.309440 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e6f4c63178440f4ba5c9f65d213df3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.313570 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.314386 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet c74e8c697d244ac2ac8174bb1fc76e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.314994 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 724d662bcf754c74a8f3dff91ee258ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:38 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.315258 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 59015204545e436cb558fe4465a67c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:38 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.316872 343 tablet_bootstrap.cc:654] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:39.317013 32055 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a3576624314d4e85b411aab9109683f7 consensus state has no leader
W20250623 08:04:39.320911 32056 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 59015204545e436cb558fe4465a67c97 (table default.Table_094f2e2ebe104774b6a713a5bbb7fd2f [id=9979c6d8878443c09e6ff9f4aea65d62], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 59015204545e436cb558fe4465a67c97 consensus state has no leader
I20250623 08:04:39.323674 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet bfd3e263464c4f2c8af9262acf211193 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f890ee0747b4df0b82bc3b396191124) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.324946 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b213c8c685c43868df1e814c79fc61f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.325320 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9f8aa424bbb4aa8a907259270d46656" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:39.325462 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5c694856f44201a8aa4314a4f81681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.325506 32655 raft_consensus.cc:3058] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.325744 32485 tablet_service.cc:1468] Processing CreateTablet for tablet a375f8ea383b44a9acd04fc2171d34ae (DEFAULT_TABLE table=default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.326107 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a375f8ea383b44a9acd04fc2171d34ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.328277 32655 raft_consensus.cc:2466] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:39.328881 979 tablet_replica.cc:331] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:39.329124 979 raft_consensus.cc:2241] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.329348 979 raft_consensus.cc:2270] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.329948 979 ts_tablet_manager.cc:1905] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.333189 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet a8945b6f49474648ac73bcb85285680a with delete_type TABLET_DATA_DELETED (replaced by tablet 1a404aac062c41e8a06207d9d8685a4d) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.336905 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b213c8c685c43868df1e814c79fc61f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.339113 339 ts_tablet_manager.cc:1428] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
I20250623 08:04:39.343662 952 ts_tablet_manager.cc:1918] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.343786 952 log.cc:1199] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/effd4fff43fe4d4fb36458986e1dfce1
I20250623 08:04:39.344051 979 ts_tablet_manager.cc:1918] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.344148 952 ts_tablet_manager.cc:1939] T effd4fff43fe4d4fb36458986e1dfce1 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:39.344493 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 138) took 5751 ms. Trace:
I20250623 08:04:39.344585 952 rpcz_store.cc:276] 0623 08:04:33.593174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:33.593232 (+ 58us) service_pool.cc:225] Handling call
0623 08:04:39.170009 (+5576777us) ts_tablet_manager.cc:1145] Deleting tablet effd4fff43fe4d4fb36458986e1dfce1
0623 08:04:39.170021 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.340704 (+170683us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.343562 (+ 2858us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.344475 (+ 913us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5575707}
I20250623 08:04:39.344921 952 tablet_replica.cc:331] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:39.345114 952 raft_consensus.cc:2241] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:39.345431 952 raft_consensus.cc:2270] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.345940 979 log.cc:1199] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/c0773fc60d4a463596f536f15d58bf57
I20250623 08:04:39.346021 952 ts_tablet_manager.cc:1905] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.346509 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a96ae9190640dea11bd11a9f0e4372 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b1a601763e44934a7db2f53d05e43a5) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.346544 979 ts_tablet_manager.cc:1939] T c0773fc60d4a463596f536f15d58bf57 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:39.346983 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet ba3d2fa01d574dd992dc904ac024ba3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.347115 32314 leader_election.cc:304] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:39.347496 1682 raft_consensus.cc:2802] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:39.347524 32489 tablet_service.cc:1468] Processing CreateTablet for tablet f20b23fd6e5c44f58b1fb2bf617c5704 (DEFAULT_TABLE table=default.Table_744bf9c83e004aafb60d2cb44530dff5 [id=dd185047529940ebbd12e27a9f6577a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.347616 1682 raft_consensus.cc:695] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:39.347846 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 275) took 1709 ms. Trace:
I20250623 08:04:39.347836 1682 consensus_queue.cc:237] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.347921 979 rpcz_store.cc:276] 0623 08:04:37.638518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.638584 (+ 66us) service_pool.cc:225] Handling call
0623 08:04:39.294612 (+1656028us) ts_tablet_manager.cc:1145] Deleting tablet c0773fc60d4a463596f536f15d58bf57
0623 08:04:39.328825 (+ 34213us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.335457 (+ 6632us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.343964 (+ 8507us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.347826 (+ 3862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653280}
W20250623 08:04:39.348798 32314 leader_election.cc:343] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: e9f8aa424bbb4aa8a907259270d46656
I20250623 08:04:39.349304 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet c0773fc60d4a463596f536f15d58bf57 (table default.Table_52d445f76863453eb994aa3b0736a5f0 [id=27968375adec4961931d1ec10e4438d8]) successfully deleted
I20250623 08:04:39.349666 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 276) took 1704 ms. Trace:
I20250623 08:04:39.350064 979 rpcz_store.cc:276] 0623 08:04:37.645090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.645263 (+ 173us) service_pool.cc:225] Handling call
0623 08:04:39.349554 (+1704291us) ts_tablet_manager.cc:1145] Deleting tablet c7fcf583064f4538913d9157e93068dd
0623 08:04:39.349565 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.349650 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1702719}
W20250623 08:04:39.350776 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet c7fcf583064f4538913d9157e93068dd because the tablet was not found. No further retry: Not found: Tablet not found: c7fcf583064f4538913d9157e93068dd
I20250623 08:04:39.351085 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 115763354f294954a260b0aa5c66a146 (DEFAULT_TABLE table=default.Table_973ac8c9b50b4fb2a8d83318384a8b76 [id=ab80a8647b644e6ebbdfba0d2e76107a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.351464 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 277) took 1706 ms. Trace:
I20250623 08:04:39.351728 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 13824916241c4cf596119d44d89cd17f (DEFAULT_TABLE table=default.Table_c491a58f8bb24bbf85189ea654bdd45f [id=826cd9624f2a4f818f02427a9462a9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.352054 979 rpcz_store.cc:276] 0623 08:04:37.645224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.645304 (+ 80us) service_pool.cc:225] Handling call
0623 08:04:39.351142 (+1705838us) ts_tablet_manager.cc:1145] Deleting tablet cdec6e9573b041e0a57d1e0e0239d97f
0623 08:04:39.351152 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.351445 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1691510}
W20250623 08:04:39.355206 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet cdec6e9573b041e0a57d1e0e0239d97f because the tablet was not found. No further retry: Not found: Tablet not found: cdec6e9573b041e0a57d1e0e0239d97f
I20250623 08:04:39.355762 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 278) took 1691 ms. Trace:
I20250623 08:04:39.356158 979 rpcz_store.cc:276] 0623 08:04:37.664597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.664681 (+ 84us) service_pool.cc:225] Handling call
0623 08:04:39.355638 (+1690957us) ts_tablet_manager.cc:1145] Deleting tablet f9f4829e103b460eb582beff3b966bb4
0623 08:04:39.355648 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.355746 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":236,"tablet-delete.queue_time_us":1690723}
W20250623 08:04:39.357801 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f9f4829e103b460eb582beff3b966bb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9f4829e103b460eb582beff3b966bb4 already in progress: creating tablet
I20250623 08:04:39.358269 32483 tablet_service.cc:1468] Processing CreateTablet for tablet 0749cb735acf4cb393ad78a2cd0c0d60 (DEFAULT_TABLE table=default.Table_7a30e48550d443ccbe36157dd0e35bcc [id=7a1f259c8fc24b4ca83cfa09e5673d58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.358708 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.358799 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet a9c8d480b8ba49a1be140b0e0a79f4e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.357371 343 tablet_bootstrap.cc:492] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:39.359072 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b213c8c685c43868df1e814c79fc61f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.359242 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a070895f4984493a3b9f6fa779797a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.359453 343 ts_tablet_manager.cc:1397] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.002s
I20250623 08:04:39.359812 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249605158634f4eab5b9d61357c383f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.360355 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet e9f8aa424bbb4aa8a907259270d46656 with delete_type TABLET_DATA_DELETED (replaced by tablet a0e9b2f810014e81a29ca9b41cd3f8ba) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.360853 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet c7fcf583064f4538913d9157e93068dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.361163 343 raft_consensus.cc:357] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.361470 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 with delete_type TABLET_DATA_DELETED (replaced by tablet db81250d1e164a7d94affd764fc0ff0c) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.361625 343 raft_consensus.cc:383] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.361815 343 raft_consensus.cc:738] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.361918 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet b80a91ae2c084020b5d3444b570b28f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.362488 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet 6926c49b787941d98c647543bd3b9ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.362354 343 consensus_queue.cc:260] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.363797 342 tablet_bootstrap.cc:492] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:39.363982 342 ts_tablet_manager.cc:1397] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.218s user 0.003s sys 0.000s
I20250623 08:04:39.364873 342 raft_consensus.cc:357] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.365096 342 raft_consensus.cc:383] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.365200 342 raft_consensus.cc:738] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.365433 342 consensus_queue.cc:260] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.367020 342 ts_tablet_manager.cc:1428] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:39.367345 342 tablet_bootstrap.cc:492] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:39.369318 342 tablet_bootstrap.cc:654] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.371841 343 ts_tablet_manager.cc:1428] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250623 08:04:39.372349 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet f02f36d6c1174a23a1f1043a431d0b6b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.372808 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet f0e24d8d890d4d1db051f172b3ec96ec with delete_type TABLET_DATA_DELETED (replaced by tablet a1cf375214c54f3491098db07faedd9f) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.373512 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 803d3607d75443568b7ddc9f0c0bdc90
I20250623 08:04:39.372877 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 279) took 1662 ms. Trace:
I20250623 08:04:39.373996 32059 catalog_manager.cc:6652] ReplaceTablet: tablet 803d3607d75443568b7ddc9f0c0bdc90 deleted and replaced by tablet a29c7fd219604bf39f40ccd09dedd2d2
I20250623 08:04:39.374641 343 tablet_bootstrap.cc:492] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:39.373898 979 rpcz_store.cc:276] 0623 08:04:37.709743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.709806 (+ 63us) service_pool.cc:225] Handling call
0623 08:04:39.370707 (+1660901us) ts_tablet_manager.cc:1145] Deleting tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7
0623 08:04:39.370718 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.372205 (+ 1487us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660547}
I20250623 08:04:39.375751 1303 raft_consensus.cc:491] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.375921 1303 raft_consensus.cc:513] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.376216 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 280) took 1664 ms. Trace:
W20250623 08:04:39.376754 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7 already in progress: creating tablet
I20250623 08:04:39.377113 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 803d3607d75443568b7ddc9f0c0bdc90 with delete_type TABLET_DATA_DELETED (replaced by tablet a29c7fd219604bf39f40ccd09dedd2d2) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.376596 979 rpcz_store.cc:276] 0623 08:04:37.711673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.711733 (+ 60us) service_pool.cc:225] Handling call
0623 08:04:39.375585 (+1663852us) ts_tablet_manager.cc:1145] Deleting tablet f83f68ebda7e4826bbb17278cd4ac5a8
0623 08:04:39.375692 (+ 107us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.376101 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1663672}
W20250623 08:04:39.377949 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f83f68ebda7e4826bbb17278cd4ac5a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f83f68ebda7e4826bbb17278cd4ac5a8 already in progress: creating tablet
I20250623 08:04:39.379525 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 284) took 1365 ms. Trace:
I20250623 08:04:39.379827 979 rpcz_store.cc:276] 0623 08:04:38.013757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.013828 (+ 71us) service_pool.cc:225] Handling call
0623 08:04:39.378663 (+1364835us) ts_tablet_manager.cc:1145] Deleting tablet 886fc0fa39c54bddb25ac01f68d95805
0623 08:04:39.378674 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.379336 (+ 662us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1364458}
W20250623 08:04:39.381079 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 886fc0fa39c54bddb25ac01f68d95805 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 886fc0fa39c54bddb25ac01f68d95805 already in progress: creating tablet
I20250623 08:04:39.382788 1303 leader_election.cc:290] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.383322 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.383616 32626 tablet_service.cc:1468] Processing CreateTablet for tablet b300341952d34600ba745245f51482d4 (DEFAULT_TABLE table=default.Table_79fce459ec304a14823e0a2179f6b804 [id=57b3f76fe53a4f69bfb544ad72a7ca52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.384088 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b300341952d34600ba745245f51482d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.385732 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a0b9402a264800a1ea39707c10da0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.386346 32703 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:39.386579 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 803d3607d75443568b7ddc9f0c0bdc90 with delete_type TABLET_DATA_DELETED (replaced by tablet a29c7fd219604bf39f40ccd09dedd2d2) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.386672 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f76ef24c69f4b17a373b242d71da8b9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.386770 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f76ef24c69f4b17a373b242d71da8b9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:39.386932 32655 raft_consensus.cc:2466] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:39.386965 32517 raft_consensus.cc:2466] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
W20250623 08:04:39.387068 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f97f14cbaffb46d681d8b11c0213cae4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f97f14cbaffb46d681d8b11c0213cae4 already in progress: creating tablet
I20250623 08:04:39.387665 339 tablet_bootstrap.cc:492] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:39.387688 32351 tablet_service.cc:1468] Processing CreateTablet for tablet b300341952d34600ba745245f51482d4 (DEFAULT_TABLE table=default.Table_79fce459ec304a14823e0a2179f6b804 [id=57b3f76fe53a4f69bfb544ad72a7ca52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.388091 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b300341952d34600ba745245f51482d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.388171 979 tablet_replica.cc:331] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:39.388849 32314 leader_election.cc:304] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:39.389412 1303 raft_consensus.cc:2802] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:39.389496 1303 raft_consensus.cc:491] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.389559 1303 raft_consensus.cc:3058] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.389734 343 tablet_bootstrap.cc:654] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.389906 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet cdec6e9573b041e0a57d1e0e0239d97f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.389633 979 raft_consensus.cc:2241] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.390720 979 raft_consensus.cc:2270] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.394392 981 ts_tablet_manager.cc:1918] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:39.394258 979 ts_tablet_manager.cc:1905] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.394567 981 log.cc:1199] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/d0b2af698b6e4ed0ae66a9bb32be7b31
W20250623 08:04:39.397794 32056 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:39501 (request call id 170) took 1642 ms (client timeout 1000 ms). Trace:
I20250623 08:04:39.399461 339 tablet_bootstrap.cc:654] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.398602 1303 raft_consensus.cc:513] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.400617 981 ts_tablet_manager.cc:1939] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:39.400874 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f76ef24c69f4b17a373b242d71da8b9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:39.401052 32655 raft_consensus.cc:3058] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.401166 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 184) took 4593 ms. Trace:
I20250623 08:04:39.401237 981 rpcz_store.cc:276] 0623 08:04:34.808043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.808084 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:39.251592 (+4443508us) ts_tablet_manager.cc:1145] Deleting tablet d0b2af698b6e4ed0ae66a9bb32be7b31
0623 08:04:39.251601 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.384182 (+132581us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.394287 (+ 10105us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.401150 (+ 6863us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4443360}
I20250623 08:04:39.401666 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet d0b2af698b6e4ed0ae66a9bb32be7b31 (table default.Table_d3f72251faee462c9c301b8aff464bb5 [id=b7671dfa27be46bb8b811b12016434e2]) successfully deleted
W20250623 08:04:39.398289 32056 rpcz_store.cc:269] 0623 08:04:37.755504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.759244 (+ 3740us) service_pool.cc:225] Handling call
0623 08:04:37.841894 (+ 82650us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f3b58 00007f19c7640176 00007f19c763fea5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:39.397639 (+1555745us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:38.426900 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:38.427026 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:38.427034 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:38.427036 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:38.427242 (+ 206us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:38.427256 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:38.427264 (+ 8us) tablet.cc:683] Acquiring locks for 4 operations
0623 08:04:38.427387 (+ 123us) tablet.cc:699] Row locks acquired
0623 08:04:38.427390 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:38.427428 (+ 38us) write_op.cc:270] Start()
0623 08:04:38.427458 (+ 30us) write_op.cc:276] Timestamp: P: 1750665878427405 usec, L: 0
0623 08:04:38.427461 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:38.427666 (+ 205us) log.cc:844] Serialized 1650 byte log entry
0623 08:04:38.430653 (+ 2987us) op_driver.cc:464] REPLICATION: finished
0623 08:04:38.430795 (+ 142us) write_op.cc:301] APPLY: starting
0623 08:04:38.430809 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:38.430870 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:38.430904 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:38.430987 (+ 83us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:38.431023 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0623 08:04:38.431031 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:38.431137 (+ 106us) log.cc:844] Serialized 41 byte log entry
0623 08:04:38.431192 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:38.431213 (+ 21us) write_op.cc:454] Released schema lock
0623 08:04:38.431219 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:38.431292 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:38.431295 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":190,"spinlock_wait_cycles":37231488,"child_traces":[["op",{"apply.queue_time_us":68,"apply.run_cpu_time_us":481,"apply.run_wall_time_us":481,"num_ops":4,"prepare.queue_time_us":2038,"prepare.run_cpu_time_us":873,"prepare.run_wall_time_us":873,"raft.queue_time_us":386,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":231,"replication_time_us":3155}]]}
I20250623 08:04:39.403195 1303 leader_election.cc:290] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.403785 32655 raft_consensus.cc:2466] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:39.403841 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f76ef24c69f4b17a373b242d71da8b9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:39.404037 32517 raft_consensus.cc:3058] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.404893 32314 leader_election.cc:304] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:39.405378 32484 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb422550efd40cf82bc92fca5c25eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.406948 32484 tablet_service.cc:1515] Processing DeleteTablet for tablet 52d0a356c74f4daea9093c998b4bf73e with delete_type TABLET_DATA_DELETED (replaced by tablet bb06a340f82047ac8ec3516edb08f4b6) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.406970 32517 raft_consensus.cc:2466] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:39.409873 32484 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.411546 1303 raft_consensus.cc:2802] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:39.411782 1303 raft_consensus.cc:695] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:39.412027 32484 tablet_service.cc:1468] Processing CreateTablet for tablet ca5c694856f44201a8aa4314a4f81681 (DEFAULT_TABLE table=default.Table_b68f23253200457683bb5cedcc56041f [id=4f81957f51bd40c192ad7c716cbd893f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.414060 1303 consensus_queue.cc:237] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.415416 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 185) took 4607 ms. Trace:
I20250623 08:04:39.415555 981 rpcz_store.cc:276] 0623 08:04:34.808417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.808458 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:39.402336 (+4593878us) ts_tablet_manager.cc:1145] Deleting tablet 2795cb14e6db4116a4c9ee539bd4e82d
0623 08:04:39.402346 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.415391 (+ 13045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4593696}
W20250623 08:04:39.416268 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 2795cb14e6db4116a4c9ee539bd4e82d because the tablet was not found. No further retry: Not found: Tablet not found: 2795cb14e6db4116a4c9ee539bd4e82d
I20250623 08:04:39.418128 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet 803d3607d75443568b7ddc9f0c0bdc90 with delete_type TABLET_DATA_DELETED (replaced by tablet a29c7fd219604bf39f40ccd09dedd2d2) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.423434 981 tablet_replica.cc:331] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:39.423686 981 raft_consensus.cc:2241] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.423854 981 raft_consensus.cc:2270] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.424435 981 ts_tablet_manager.cc:1905] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.427923 32059 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_f7811b4c393d498cbf95abe8a8cdcd38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:39.428650 32059 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7811b4c393d498cbf95abe8a8cdcd38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.429214 342 tablet_bootstrap.cc:492] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
W20250623 08:04:39.366313 32055 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.1:45613 (request call id 157) took 1169 ms (client timeout 1000 ms). Trace:
I20250623 08:04:39.432303 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet f279b63b55c64e548ded27b02025c9f5 (table default.Table_6e40e423312e4f9e95174f277b8897ce [id=7038d3807ece414f8a8e76bfe107c3b3]) successfully deleted
I20250623 08:04:39.433323 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 358cb6ac24414bdfbc32155e732f9ac8 (table default.Table_06b64498b6e148d5b86187f2ce7e4c31 [id=b827ea08f0b14c9aa2b2c3d02d15f150]) successfully deleted
I20250623 08:04:39.435925 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 7f890ee0747b4df0b82bc3b396191124 (DEFAULT_TABLE table=default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 [id=9f97f6deca194137a353ea221d0e3676]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.436086 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 115763354f294954a260b0aa5c66a146. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.436344 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f890ee0747b4df0b82bc3b396191124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.436404 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0749cb735acf4cb393ad78a2cd0c0d60. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:39.432358 32055 rpcz_store.cc:269] 0623 08:04:38.196822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.412383 (+215561us) service_pool.cc:225] Handling call
0623 08:04:39.366177 (+953794us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:38.841482 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:38.841607 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:38.841614 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:38.841617 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:38.841736 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:38.841750 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:38.841755 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:38.841818 (+ 63us) tablet.cc:699] Row locks acquired
0623 08:04:38.841820 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:38.841857 (+ 37us) write_op.cc:270] Start()
0623 08:04:38.841891 (+ 34us) write_op.cc:276] Timestamp: P: 1750665878841790 usec, L: 0
0623 08:04:38.841894 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:38.842117 (+ 223us) log.cc:844] Serialized 527 byte log entry
0623 08:04:38.860903 (+ 18786us) op_driver.cc:464] REPLICATION: finished
0623 08:04:38.861069 (+ 166us) write_op.cc:301] APPLY: starting
0623 08:04:38.861087 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:38.861163 (+ 76us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:38.861242 (+ 79us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:38.861418 (+ 176us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:38.861574 (+ 156us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:38.861583 (+ 9us) write_op.cc:312] APPLY: finished
0623 08:04:38.861769 (+ 186us) log.cc:844] Serialized 23 byte log entry
0623 08:04:38.861861 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:38.861912 (+ 51us) write_op.cc:454] Released schema lock
0623 08:04:38.861920 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:38.862294 (+ 374us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:38.862299 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":761,"reported_tablets":193,"child_traces":[["op",{"apply.queue_time_us":85,"apply.run_cpu_time_us":578,"apply.run_wall_time_us":1210,"num_ops":1,"prepare.queue_time_us":17178,"prepare.run_cpu_time_us":753,"prepare.run_wall_time_us":755,"raft.queue_time_us":9676,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":247,"replication_time_us":18974}]]}
I20250623 08:04:39.443336 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13824916241c4cf596119d44d89cd17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.447119 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f20b23fd6e5c44f58b1fb2bf617c5704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.448071 32053 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 90ceb0f8d5c94c4e9427042d8a35e16c
I20250623 08:04:39.448995 979 ts_tablet_manager.cc:1918] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.449287 979 log.cc:1199] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/025365d2fa0a43dd983d395a05cc2af1
I20250623 08:04:39.449841 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 90ceb0f8d5c94c4e9427042d8a35e16c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.449901 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 90ceb0f8d5c94c4e9427042d8a35e16c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.450373 979 ts_tablet_manager.cc:1939] T 025365d2fa0a43dd983d395a05cc2af1 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:39.451088 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5c694856f44201a8aa4314a4f81681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.435045 342 ts_tablet_manager.cc:1397] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.002s
I20250623 08:04:39.459656 342 raft_consensus.cc:357] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.459722 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 7f890ee0747b4df0b82bc3b396191124 (DEFAULT_TABLE table=default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 [id=9f97f6deca194137a353ea221d0e3676]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.459878 342 raft_consensus.cc:383] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.459975 342 raft_consensus.cc:738] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.460227 342 consensus_queue.cc:260] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.460626 32053 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_1a1b8de8d0a645e9a48938956c562be9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:39.461282 342 ts_tablet_manager.cc:1428] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250623 08:04:39.461241 32053 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a1b8de8d0a645e9a48938956c562be9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.461633 342 tablet_bootstrap.cc:492] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:39.466408 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 9049e3d9b39f420bb51221808409fc54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9049e3d9b39f420bb51221808409fc54 already in progress: creating tablet
I20250623 08:04:39.468147 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e602d391526d4b0eb0dbf5189afbe075" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:39.468336 32373 raft_consensus.cc:3058] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.471180 32373 raft_consensus.cc:2466] T e602d391526d4b0eb0dbf5189afbe075 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:39.471701 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet dc0dad440b494018bc0d6cfd40940bc1 (table default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]) successfully deleted
I20250623 08:04:39.472115 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e602d391526d4b0eb0dbf5189afbe075" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:39.472278 32649 raft_consensus.cc:3058] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.475921 981 ts_tablet_manager.cc:1918] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.476089 981 log.cc:1199] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/d6ed4cb252b1452087f8ee53ec78d985
I20250623 08:04:39.476507 981 ts_tablet_manager.cc:1939] T d6ed4cb252b1452087f8ee53ec78d985 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:39.477571 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 485538f7c63948818f20a25c02381519 (table default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]) successfully deleted
I20250623 08:04:39.478355 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e9bf7cba2244703b7402cb4be1a005e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:39.478562 32373 raft_consensus.cc:3058] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.481333 32373 raft_consensus.cc:2466] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:39.481469 32649 raft_consensus.cc:2466] T e602d391526d4b0eb0dbf5189afbe075 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:39.487718 339 tablet_bootstrap.cc:492] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:39.487908 339 ts_tablet_manager.cc:1397] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.107s user 0.006s sys 0.007s
I20250623 08:04:39.488695 339 raft_consensus.cc:357] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.489337 339 raft_consensus.cc:383] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.489426 339 raft_consensus.cc:738] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.489673 339 consensus_queue.cc:260] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.490242 339 ts_tablet_manager.cc:1428] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:39.490569 339 tablet_bootstrap.cc:492] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:39.490880 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e9bf7cba2244703b7402cb4be1a005e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:39.491086 32649 raft_consensus.cc:3058] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.491093 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87648b11b10b45fc9cc8c998bbceae92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.492090 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 8e2a562010944837b4e486e0facd47cf (DEFAULT_TABLE table=default.Table_334749aabade478fbbc2badb1e4960bf [id=ec79bae33a55429fb05f25780369b398]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.492496 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2a562010944837b4e486e0facd47cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.495051 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 186) took 4686 ms. Trace:
I20250623 08:04:39.495162 981 rpcz_store.cc:276] 0623 08:04:34.808815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.808858 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:39.423374 (+4614516us) ts_tablet_manager.cc:1145] Deleting tablet d6ed4cb252b1452087f8ee53ec78d985
0623 08:04:39.423387 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.430074 (+ 6687us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.475815 (+ 45741us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.495030 (+ 19215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4614353}
I20250623 08:04:39.497352 981 tablet_replica.cc:331] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:39.498155 981 raft_consensus.cc:2241] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.498909 981 raft_consensus.cc:2270] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.493825 32649 raft_consensus.cc:2466] T 0e9bf7cba2244703b7402cb4be1a005e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:39.503033 981 ts_tablet_manager.cc:1905] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.509857 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet d6ed4cb252b1452087f8ee53ec78d985 (table default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]) successfully deleted
I20250623 08:04:39.515988 343 tablet_bootstrap.cc:492] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:39.516544 343 ts_tablet_manager.cc:1397] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.142s user 0.001s sys 0.023s
I20250623 08:04:39.515962 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 8e2a562010944837b4e486e0facd47cf (DEFAULT_TABLE table=default.Table_334749aabade478fbbc2badb1e4960bf [id=ec79bae33a55429fb05f25780369b398]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.526408 339 tablet_bootstrap.cc:654] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:39.529218 32051 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16bc969d09f74b0986070a504f9f670e consensus state has no leader
I20250623 08:04:39.529428 1627 raft_consensus.cc:491] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.529564 1627 raft_consensus.cc:513] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.530129 1627 leader_election.cc:290] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:39.522296 343 raft_consensus.cc:357] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.530652 343 raft_consensus.cc:383] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.530791 343 raft_consensus.cc:738] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.531054 343 consensus_queue.cc:260] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.531143 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_905d0330c27149a19dabb586c51a2e0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:39.531724 343 ts_tablet_manager.cc:1428] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.015s user 0.000s sys 0.001s
W20250623 08:04:39.531874 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_905d0330c27149a19dabb586c51a2e0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.533483 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f890ee0747b4df0b82bc3b396191124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.533826 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 298) took 1087 ms. Trace:
I20250623 08:04:39.533915 979 rpcz_store.cc:276] 0623 08:04:38.446520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.452187 (+ 5667us) service_pool.cc:225] Handling call
0623 08:04:39.387806 (+935619us) ts_tablet_manager.cc:1145] Deleting tablet 025365d2fa0a43dd983d395a05cc2af1
0623 08:04:39.387816 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.416891 (+ 29075us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.448115 (+ 31224us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.533806 (+ 85691us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":935328}
I20250623 08:04:39.534220 979 tablet_replica.cc:331] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:39.534404 979 raft_consensus.cc:2241] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.534524 979 raft_consensus.cc:2270] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.535001 979 ts_tablet_manager.cc:1905] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.538558 342 tablet_bootstrap.cc:654] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.539321 32058 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_2ecc70ddf2dc40e081fefb676d46a68d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:39.539929 32058 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ecc70ddf2dc40e081fefb676d46a68d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.541429 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e851e10456a04a51814a556167f4777f" } new_table_name: "default.Table_0b9c0d17ddcd422eb8035e69884f3da6" modify_external_catalogs: true
I20250623 08:04:39.541566 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f76ef24c69f4b17a373b242d71da8b9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.543020 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_f0683327c2bf468582a2f3e557d15725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:39.543537 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0683327c2bf468582a2f3e557d15725 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.543922 1627 raft_consensus.cc:491] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.544098 1627 raft_consensus.cc:513] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.544267 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b405f87bb1e4d0686a1f627835d901f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:39.544391 32515 raft_consensus.cc:2466] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:39.544631 1627 leader_election.cc:290] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:39.544682 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38eecf4a94d4536b98d4f6e037a0094" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:39.544836 32515 raft_consensus.cc:2466] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:39.544936 32057 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_848fa16d96984425bfb59432e4b51723" } modify_external_catalogs: true
I20250623 08:04:39.545084 32057 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_848fa16d96984425bfb59432e4b51723" } modify_external_catalogs: true
I20250623 08:04:39.546468 32059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_54fe37774d62482296afe3191f2e8e88" } modify_external_catalogs: true
I20250623 08:04:39.546649 32059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_54fe37774d62482296afe3191f2e8e88" } modify_external_catalogs: true
I20250623 08:04:39.541760 32649 raft_consensus.cc:2391] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:39.548064 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861e45cf546487d9812caa06e2704eb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:39.548218 32517 raft_consensus.cc:3058] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:39.548486 32453 consensus_peers.cc:487] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.550935 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7874a1d5cb9a464294ca32b2767efbf3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:39.551122 32519 raft_consensus.cc:2466] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
W20250623 08:04:39.551358 32453 leader_election.cc:343] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 485538f7c63948818f20a25c02381519
I20250623 08:04:39.551389 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73453eb1a0394cfe8d49adcc71d4f3e9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:39.551481 32453 leader_election.cc:304] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
W20250623 08:04:39.551666 32453 raft_consensus.cc:2684] T 485538f7c63948818f20a25c02381519 P 59f7c614254f42009306b13337f06576: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:39.552069 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a96ae9190640dea11bd11a9f0e4372" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:39.553390 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_8ab4ba6029964c2fa123202d6c5c836d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:39.553969 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ab4ba6029964c2fa123202d6c5c836d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:39.554910 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87648b11b10b45fc9cc8c998bbceae92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.555018 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:39.555092 32646 raft_consensus.cc:2466] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:39.557462 32517 raft_consensus.cc:2466] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:39.557621 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f76ef24c69f4b17a373b242d71da8b9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.543015 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 025365d2fa0a43dd983d395a05cc2af1 (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]) successfully deleted
I20250623 08:04:39.558200 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet effd4fff43fe4d4fb36458986e1dfce1 (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]) successfully deleted
I20250623 08:04:39.559047 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2a562010944837b4e486e0facd47cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:39.560424 32314 consensus_peers.cc:487] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c01121f2e644d6aa6e8ec228bf75be9. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.566403 979 ts_tablet_manager.cc:1918] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:39.566596 979 log.cc:1199] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/724d662bcf754c74a8f3dff91ee258ae
I20250623 08:04:39.566999 979 ts_tablet_manager.cc:1939] T 724d662bcf754c74a8f3dff91ee258ae P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:39.567198 952 ts_tablet_manager.cc:1918] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.567353 952 log.cc:1199] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/8ffafd10f26740c9a5b6c9bdc3bb23a0
I20250623 08:04:39.567756 952 ts_tablet_manager.cc:1939] T 8ffafd10f26740c9a5b6c9bdc3bb23a0 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:39.568073 343 tablet_bootstrap.cc:492] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:39.568095 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 158) took 5113 ms. Trace:
I20250623 08:04:39.568184 952 rpcz_store.cc:276] 0623 08:04:34.455020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.457435 (+ 2415us) service_pool.cc:225] Handling call
0623 08:04:39.344878 (+4887443us) ts_tablet_manager.cc:1145] Deleting tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0
0623 08:04:39.344888 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.535758 (+190870us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.567100 (+ 31342us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.568081 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4886588}
I20250623 08:04:39.569931 343 tablet_bootstrap.cc:654] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.567785 32568 heartbeater.cc:344] Connected to a master server at 127.31.69.61:33313
I20250623 08:04:39.579062 952 tablet_replica.cc:331] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:39.580459 342 tablet_bootstrap.cc:492] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:39.581003 342 ts_tablet_manager.cc:1397] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.001s
I20250623 08:04:39.586238 342 raft_consensus.cc:357] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.586995 342 raft_consensus.cc:383] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.587195 342 raft_consensus.cc:738] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.587723 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 724d662bcf754c74a8f3dff91ee258ae (table default.Table_9fc75fd22c6a467782fe3b4c3b123602 [id=1a49c0c47b6647ef8fc4c7c20ad9bc1b]) successfully deleted
I20250623 08:04:39.588296 342 consensus_queue.cc:260] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.589437 342 ts_tablet_manager.cc:1428] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250623 08:04:39.590332 342 tablet_bootstrap.cc:492] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:39.592590 32453 leader_election.cc:343] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.593011 32453 leader_election.cc:304] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
W20250623 08:04:39.593379 32453 raft_consensus.cc:2684] T dc0dad440b494018bc0d6cfd40940bc1 P 59f7c614254f42009306b13337f06576: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:39.580660 952 raft_consensus.cc:2241] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.595577 952 raft_consensus.cc:2270] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.596154 952 ts_tablet_manager.cc:1905] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:39.597162 32453 leader_election.cc:343] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:39.598469 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet c74e8c697d244ac2ac8174bb1fc76e61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c74e8c697d244ac2ac8174bb1fc76e61 already in progress: creating tablet
W20250623 08:04:39.598912 32044 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 89a24019b797467e857b744d701fb1ed consensus state has no leader
W20250623 08:04:39.602596 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 59015204545e436cb558fe4465a67c97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59015204545e436cb558fe4465a67c97 already in progress: creating tablet
I20250623 08:04:39.591869 981 ts_tablet_manager.cc:1918] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:39.599009 339 tablet_bootstrap.cc:492] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:39.604549 981 log.cc:1199] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/70f9d21b8c144f9cada9bdbe0fbb1538
I20250623 08:04:39.604902 339 ts_tablet_manager.cc:1397] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.000s
I20250623 08:04:39.605335 981 ts_tablet_manager.cc:1939] T 70f9d21b8c144f9cada9bdbe0fbb1538 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:39.606364 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 187) took 4797 ms. Trace:
I20250623 08:04:39.606467 981 rpcz_store.cc:276] 0623 08:04:34.809334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.809376 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:39.497298 (+4687922us) ts_tablet_manager.cc:1145] Deleting tablet 70f9d21b8c144f9cada9bdbe0fbb1538
0623 08:04:39.497310 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.507311 (+ 10001us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.591758 (+ 84447us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.606345 (+ 14587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4687745}
I20250623 08:04:39.607245 979 tablet_replica.cc:331] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:39.607481 979 raft_consensus.cc:2241] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.607673 979 raft_consensus.cc:2270] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.608130 979 ts_tablet_manager.cc:1905] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.608454 32453 leader_election.cc:304] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.611229 1627 raft_consensus.cc:2747] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.611706 32453 leader_election.cc:304] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.611871 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 70f9d21b8c144f9cada9bdbe0fbb1538 (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]) successfully deleted
I20250623 08:04:39.594696 342 tablet_bootstrap.cc:654] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.614128 1627 raft_consensus.cc:2747] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:39.614703 32453 leader_election.cc:343] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.607012 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 188) took 4797 ms. Trace:
I20250623 08:04:39.612756 339 raft_consensus.cc:357] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.617280 1303 raft_consensus.cc:491] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.617136 981 rpcz_store.cc:276] 0623 08:04:34.809740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.809784 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:39.606893 (+4797109us) ts_tablet_manager.cc:1145] Deleting tablet 6926c49b787941d98c647543bd3b9ed5
0623 08:04:39.606903 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.606994 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4796957}
I20250623 08:04:39.617282 339 raft_consensus.cc:383] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.617630 1303 raft_consensus.cc:513] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.618183 1303 leader_election.cc:290] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.618139 339 raft_consensus.cc:738] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:39.618219 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 6926c49b787941d98c647543bd3b9ed5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6926c49b787941d98c647543bd3b9ed5 already in progress: creating tablet
I20250623 08:04:39.619886 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52d0a356c74f4daea9093c998b4bf73e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.620121 32655 raft_consensus.cc:2466] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:39.620216 339 consensus_queue.cc:260] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.621912 32314 leader_election.cc:304] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:39.622203 32314 leader_election.cc:343] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 52d0a356c74f4daea9093c998b4bf73e
I20250623 08:04:39.623870 1303 raft_consensus.cc:2802] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:39.623948 1303 raft_consensus.cc:491] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.624007 1303 raft_consensus.cc:3058] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.626637 1303 raft_consensus.cc:513] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.627486 1303 leader_election.cc:290] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:39.628692 32314 leader_election.cc:343] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 52d0a356c74f4daea9093c998b4bf73e
I20250623 08:04:39.628506 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52d0a356c74f4daea9093c998b4bf73e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:39.628952 32655 raft_consensus.cc:3058] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.630117 1659 consensus_queue.cc:1035] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:39.631812 32655 raft_consensus.cc:2466] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:39.632248 32314 leader_election.cc:304] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:39.632484 1303 raft_consensus.cc:2802] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:39.632586 1303 raft_consensus.cc:695] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:39.632794 32453 leader_election.cc:304] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:39.633011 981 tablet_replica.cc:331] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:39.633129 1507 raft_consensus.cc:2802] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:39.633258 1507 raft_consensus.cc:695] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:39.633157 1303 consensus_queue.cc:237] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.633488 1507 consensus_queue.cc:237] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.635874 32453 leader_election.cc:304] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:39.636107 1716 raft_consensus.cc:2802] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:39.636214 1716 raft_consensus.cc:695] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
W20250623 08:04:39.636581 32453 leader_election.cc:343] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:39.637676 32453 consensus_peers.cc:487] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6ed4cb252b1452087f8ee53ec78d985. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.638043 343 tablet_bootstrap.cc:492] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:39.638219 343 ts_tablet_manager.cc:1397] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.002s
I20250623 08:04:39.638247 1627 consensus_queue.cc:1035] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:39.639015 343 raft_consensus.cc:357] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.640934 343 raft_consensus.cc:383] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.641053 343 raft_consensus.cc:738] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.641353 343 consensus_queue.cc:260] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.641611 339 ts_tablet_manager.cc:1428] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.036s user 0.004s sys 0.000s
I20250623 08:04:39.644789 343 ts_tablet_manager.cc:1428] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250623 08:04:39.645150 343 tablet_bootstrap.cc:492] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:39.645792 1707 raft_consensus.cc:491] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.645906 1707 raft_consensus.cc:513] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.646600 343 tablet_bootstrap.cc:654] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.647578 1707 leader_election.cc:290] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.647297 339 tablet_bootstrap.cc:492] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:39.648180 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9d1c8a67e149478a0d69842a4eb5b2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:39.648659 32589 leader_election.cc:343] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.652143 981 raft_consensus.cc:2241] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:39.652431 981 pending_rounds.cc:70] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:39.652549 981 pending_rounds.cc:77] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727442980196352 op_type: NO_OP noop_request { }
I20250623 08:04:39.652696 981 raft_consensus.cc:2887] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:39.652832 981 raft_consensus.cc:2270] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.653635 981 ts_tablet_manager.cc:1905] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.655340 1716 consensus_queue.cc:237] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.666257 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7874a1d5cb9a464294ca32b2767efbf3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.667917 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38eecf4a94d4536b98d4f6e037a0094" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.668287 32589 leader_election.cc:304] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.668709 1707 raft_consensus.cc:2802] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:39.668787 1707 raft_consensus.cc:491] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.668848 1707 raft_consensus.cc:3058] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.671473 1707 raft_consensus.cc:513] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.672580 1707 leader_election.cc:290] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.674067 32589 leader_election.cc:304] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.675180 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b405f87bb1e4d0686a1f627835d901f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:39.676851 32589 leader_election.cc:304] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.682853 979 ts_tablet_manager.cc:1918] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:39.682964 979 log.cc:1199] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/bfd3e263464c4f2c8af9262acf211193
I20250623 08:04:39.683275 979 ts_tablet_manager.cc:1939] T bfd3e263464c4f2c8af9262acf211193 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:39.683337 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 90ceb0f8d5c94c4e9427042d8a35e16c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.687250 1707 raft_consensus.cc:2802] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:39.687444 1707 raft_consensus.cc:491] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.687629 1707 raft_consensus.cc:3058] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.690357 1707 raft_consensus.cc:513] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.691118 1707 leader_election.cc:290] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.706076 1707 raft_consensus.cc:2802] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:39.706271 1707 raft_consensus.cc:491] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.706439 1707 raft_consensus.cc:3058] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.712975 32484 tablet_service.cc:1468] Processing CreateTablet for tablet b300341952d34600ba745245f51482d4 (DEFAULT_TABLE table=default.Table_79fce459ec304a14823e0a2179f6b804 [id=57b3f76fe53a4f69bfb544ad72a7ca52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.713402 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b300341952d34600ba745245f51482d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:39.714111 32453 consensus_peers.cc:487] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 136e482995a24c5daec55a701f3091fa. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.714475 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 8e2a562010944837b4e486e0facd47cf (DEFAULT_TABLE table=default.Table_334749aabade478fbbc2badb1e4960bf [id=ec79bae33a55429fb05f25780369b398]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.714876 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e2a562010944837b4e486e0facd47cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.714476 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 7f890ee0747b4df0b82bc3b396191124 (DEFAULT_TABLE table=default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 [id=9f97f6deca194137a353ea221d0e3676]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:39.716899 32453 consensus_peers.cc:487] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a0bbafe1ea34ab5ae1b3de25b50d4a3. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.717147 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f890ee0747b4df0b82bc3b396191124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.717507 1508 consensus_queue.cc:1035] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:39.717202 32453 consensus_peers.cc:487] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0773fc60d4a463596f536f15d58bf57. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.717615 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 (table default.Table_3ef2e86e99164cce8e882e6954e2ec17 [id=dbbd669a72954d1aaf169b5870a85fc3]) successfully deleted
W20250623 08:04:39.720649 32453 consensus_peers.cc:487] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48da1e4759bb4fea87a45ad8a428b6e7. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:39.721344 32453 consensus_peers.cc:487] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 283bebc7063d437ba3e541d7dee3c0b5. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:39.721899 32453 consensus_peers.cc:487] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26f4a1b472e848ccb9a1ec71c1dd3516. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.722191 32453 leader_election.cc:304] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.722697 1507 raft_consensus.cc:2747] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.722659 1626 consensus_queue.cc:1035] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:39.723465 952 ts_tablet_manager.cc:1918] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.723564 952 log.cc:1199] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/cdec6e9573b041e0a57d1e0e0239d97f
I20250623 08:04:39.725016 32453 leader_election.cc:304] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.725587 1659 raft_consensus.cc:2802] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:39.725661 1659 raft_consensus.cc:491] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:39.725858 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9d1c8a67e149478a0d69842a4eb5b2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:39.727838 952 ts_tablet_manager.cc:1939] T cdec6e9573b041e0a57d1e0e0239d97f P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:39.728062 32453 leader_election.cc:343] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.728770 1508 consensus_queue.cc:1035] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:39.732209 32057 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet aface2a39d764a6cb6e4965867ab7b38
I20250623 08:04:39.734699 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b405f87bb1e4d0686a1f627835d901f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:39.735424 32050 master_service.cc:948] ReplaceTablet: received request to replace tablet 9dbbdbebdf3b4b50bda31a94fc0de36f from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:39.736464 32054 master_service.cc:948] ReplaceTablet: received request to replace tablet 30dfa986dfd44658a0ffb07796030520 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:39.737411 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 1aa604f500984e3b969e15592cdbed6f
I20250623 08:04:39.745126 979 tablet_replica.cc:331] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:39.745416 979 raft_consensus.cc:2241] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.745640 979 raft_consensus.cc:2270] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.746299 979 ts_tablet_manager.cc:1905] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:39.761543 32453 leader_election.cc:343] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:39.762712 32453 consensus_peers.cc:487] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4a0b9402a264800a1ea39707c10da0c. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:39.763808 32059 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6bb7361aabf7440589a3958da502f36c" } modify_external_catalogs: true
I20250623 08:04:39.764004 32059 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_6bb7361aabf7440589a3958da502f36c" } modify_external_catalogs: true
I20250623 08:04:39.764561 32514 raft_consensus.cc:3058] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.765363 1659 raft_consensus.cc:3058] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.765822 32056 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_ee8a2c2391624131a4aac14a90311aa0" } new_table_name: "default.Table_d2473e1e63c84b229243a1c676266ca6" modify_external_catalogs: true
I20250623 08:04:39.766206 1508 raft_consensus.cc:491] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.766346 1508 raft_consensus.cc:513] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.767325 32053 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_739315d3797c465ca4ac459ddc3477e5" } new_table_name: "default.Table_6ac4b9cba3e1422faf62f4002fb70452" modify_external_catalogs: true
I20250623 08:04:39.768520 32058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fbb1fc5910454112b795f72c26f1af5b" } modify_external_catalogs: true
I20250623 08:04:39.768573 1508 leader_election.cc:290] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:39.768673 32058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fbb1fc5910454112b795f72c26f1af5b" } modify_external_catalogs: true
I20250623 08:04:39.768980 1780 raft_consensus.cc:491] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.769145 1780 raft_consensus.cc:513] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.769846 1780 leader_election.cc:290] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.769884 32057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_56d841d4028b4bcf9282392e22379235" } new_table_name: "default.Table_faafe8f5734d448da0da414ad0c4e9ac" modify_external_catalogs: true
I20250623 08:04:39.775300 1659 raft_consensus.cc:513] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.776371 1720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d31835ec4ad74332a00998d431492466. Latest consensus state: current_term: 3 leader_uuid: "d31835ec4ad74332a00998d431492466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:39.776657 1720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:39.776845 1720 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d31835ec4ad74332a00998d431492466)
I20250623 08:04:39.777065 1720 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:39.778472 1659 leader_election.cc:290] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:39.779234 1764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d31835ec4ad74332a00998d431492466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:39.779573 1764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:39.780575 32514 raft_consensus.cc:2466] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:39.783970 342 tablet_bootstrap.cc:492] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:39.784226 342 ts_tablet_manager.cc:1397] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.194s user 0.014s sys 0.006s
I20250623 08:04:39.785422 1720 leader_election.cc:290] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:39.786202 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 157) took 5330 ms. Trace:
I20250623 08:04:39.786343 952 rpcz_store.cc:276] 0623 08:04:34.454921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.458933 (+ 4012us) service_pool.cc:225] Handling call
0623 08:04:39.568462 (+5109529us) ts_tablet_manager.cc:1145] Deleting tablet cdec6e9573b041e0a57d1e0e0239d97f
0623 08:04:39.568472 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.632366 (+ 63894us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.723367 (+ 91001us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.785682 (+ 62315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5107343}
I20250623 08:04:39.786984 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 160) took 5328 ms. Trace:
I20250623 08:04:39.787076 952 rpcz_store.cc:276] 0623 08:04:34.458181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.459924 (+ 1743us) service_pool.cc:225] Handling call
0623 08:04:39.786606 (+5326682us) ts_tablet_manager.cc:1145] Deleting tablet 92005abc61dd40ef9da8124cadc29b6a
0623 08:04:39.786615 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.786730 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5324426}
I20250623 08:04:39.787351 952 tablet_replica.cc:331] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:39.787550 952 raft_consensus.cc:2241] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.787741 952 raft_consensus.cc:2270] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.788323 952 ts_tablet_manager.cc:1905] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.791373 342 raft_consensus.cc:357] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.791669 342 raft_consensus.cc:383] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.763041 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet bfd3e263464c4f2c8af9262acf211193 (table default.Table_c7119eddeaf44e1b89d2dbd1e38151f7 [id=9f97f6deca194137a353ea221d0e3676]) successfully deleted
W20250623 08:04:39.794263 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet a8945b6f49474648ac73bcb85285680a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8945b6f49474648ac73bcb85285680a already in progress: creating tablet
I20250623 08:04:39.795200 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet aface2a39d764a6cb6e4965867ab7b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.795459 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 1aa604f500984e3b969e15592cdbed6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.797052 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet cdec6e9573b041e0a57d1e0e0239d97f (table default.Table_48cbdc58fd034e6390087ed7c6346d24 [id=ef1a8122b8044388b4ae24a03da26837]) successfully deleted
W20250623 08:04:39.797324 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 92005abc61dd40ef9da8124cadc29b6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92005abc61dd40ef9da8124cadc29b6a already in progress: creating tablet
I20250623 08:04:39.797504 32484 tablet_service.cc:1515] Processing DeleteTablet for tablet aface2a39d764a6cb6e4965867ab7b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.811918 343 tablet_bootstrap.cc:492] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:39.812180 343 ts_tablet_manager.cc:1397] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.167s user 0.006s sys 0.003s
I20250623 08:04:39.813115 343 raft_consensus.cc:357] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.813427 343 raft_consensus.cc:383] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.813558 343 raft_consensus.cc:738] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.814056 343 consensus_queue.cc:260] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.816363 1267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55636 (request call id 0) took 4967 ms. Trace:
I20250623 08:04:39.819649 32644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38eecf4a94d4536b98d4f6e037a0094" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.819927 32644 raft_consensus.cc:2391] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:39.820199 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7874a1d5cb9a464294ca32b2767efbf3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.820765 981 ts_tablet_manager.cc:1918] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:39.820961 981 log.cc:1199] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/52d0a356c74f4daea9093c998b4bf73e
W20250623 08:04:39.821353 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 144507us
W20250623 08:04:39.821635 32589 leader_election.cc:343] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.821836 32589 leader_election.cc:304] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:39.822319 32589 leader_election.cc:343] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.822496 32589 leader_election.cc:304] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.822947 32589 leader_election.cc:304] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.710716 1707 raft_consensus.cc:513] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.824091 1707 leader_election.cc:290] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:39.824273 1707 raft_consensus.cc:2747] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.824565 1707 consensus_queue.cc:1035] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:39.824972 1707 raft_consensus.cc:2747] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.825264 1707 raft_consensus.cc:2802] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:39.825398 1707 raft_consensus.cc:695] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:39.825647 1707 consensus_queue.cc:237] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.828351 981 ts_tablet_manager.cc:1939] T 52d0a356c74f4daea9093c998b4bf73e P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:39.828861 32655 raft_consensus.cc:2391] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:39.829846 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 1aa604f500984e3b969e15592cdbed6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.829881 342 raft_consensus.cc:738] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.830212 342 consensus_queue.cc:260] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.832698 1784 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250623 08:04:39.836540 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42c861e29e1e4e549021dae7255a856c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:39.836881 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87648b11b10b45fc9cc8c998bbceae92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:39.837282 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet aface2a39d764a6cb6e4965867ab7b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.837428 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 1aa604f500984e3b969e15592cdbed6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.840885 342 ts_tablet_manager.cc:1428] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.047s user 0.001s sys 0.001s
I20250623 08:04:39.842669 952 ts_tablet_manager.cc:1918] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:39.842840 952 log.cc:1199] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a25bdc9b7e48467fba3771365e58bcbc
I20250623 08:04:39.843626 952 ts_tablet_manager.cc:1939] T a25bdc9b7e48467fba3771365e58bcbc P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:39.844009 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 156) took 5389 ms. Trace:
I20250623 08:04:39.844555 952 rpcz_store.cc:276] 0623 08:04:34.454781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.459873 (+ 5092us) service_pool.cc:225] Handling call
0623 08:04:39.787269 (+5327396us) ts_tablet_manager.cc:1145] Deleting tablet a25bdc9b7e48467fba3771365e58bcbc
0623 08:04:39.787277 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.792802 (+ 5525us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.809944 (+ 17142us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.843969 (+ 34025us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5324465}
I20250623 08:04:39.845023 952 tablet_replica.cc:331] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:39.845347 952 raft_consensus.cc:2241] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:39.845570 952 raft_consensus.cc:2270] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.846262 342 tablet_bootstrap.cc:492] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:39.847046 952 ts_tablet_manager.cc:1905] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:39.848511 32453 consensus_peers.cc:487] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0b2af698b6e4ed0ae66a9bb32be7b31. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:39.848774 32453 leader_election.cc:343] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:39.849545 32453 leader_election.cc:343] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.853924 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet a25bdc9b7e48467fba3771365e58bcbc (table default.Table_6648aba827f24ba29ee9d94ee7c96089 [id=149914e63b704358866228fb74895591]) successfully deleted
I20250623 08:04:39.858122 342 tablet_bootstrap.cc:654] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.858886 32051 master_service.cc:948] ReplaceTablet: received request to replace tablet 185c2bf7b9254947bb76b7fe0924559e from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:39.864368 1627 consensus_queue.cc:1035] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:39.867487 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 189) took 5057 ms. Trace:
I20250623 08:04:39.867805 981 rpcz_store.cc:276] 0623 08:04:34.810133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.810171 (+ 38us) service_pool.cc:225] Handling call
0623 08:04:39.632971 (+4822800us) ts_tablet_manager.cc:1145] Deleting tablet 52d0a356c74f4daea9093c998b4bf73e
0623 08:04:39.632979 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:39.656877 (+ 23898us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.820617 (+163740us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:39.867472 (+ 46855us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18851,"tablet-delete.queue_time_us":4822629}
I20250623 08:04:39.868628 981 tablet_replica.cc:331] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:39.877532 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 52d0a356c74f4daea9093c998b4bf73e (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:39.882222 981 raft_consensus.cc:2241] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:39.882491 981 raft_consensus.cc:2270] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:39.883150 981 ts_tablet_manager.cc:1905] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:39.888221 1778 tablet.cc:1722] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:39.889743 1777 tablet.cc:1722] T 7419cb91360f49e1b3cb7e90732558c5 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:39.898154 32051 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 185c2bf7b9254947bb76b7fe0924559e
I20250623 08:04:39.898784 32051 catalog_manager.cc:6652] ReplaceTablet: tablet 185c2bf7b9254947bb76b7fe0924559e deleted and replaced by tablet 04fde7d1c1744c0597e43086ca4c9541
I20250623 08:04:39.904778 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 185c2bf7b9254947bb76b7fe0924559e with delete_type TABLET_DATA_DELETED (replaced by tablet 04fde7d1c1744c0597e43086ca4c9541) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.908757 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42c861e29e1e4e549021dae7255a856c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:39.908946 32649 raft_consensus.cc:2466] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:39.909533 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87648b11b10b45fc9cc8c998bbceae92" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:39.909658 32649 raft_consensus.cc:3058] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.912410 32649 raft_consensus.cc:2466] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:39.913972 1245 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7351aa31800245659edc9837e27daffd" instance_seqno: 1750665860475404) as {username='slave'} at 127.31.69.1:56523; Asking this server to re-register.
I20250623 08:04:39.918303 342 tablet_bootstrap.cc:492] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:39.918470 342 ts_tablet_manager.cc:1397] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.003s
I20250623 08:04:39.924870 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 185c2bf7b9254947bb76b7fe0924559e with delete_type TABLET_DATA_DELETED (replaced by tablet 04fde7d1c1744c0597e43086ca4c9541) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.927616 1777 tablet.cc:1722] T a1c82e4d2db64026a04c5c84bc1e09ec P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:39.927727 1778 tablet.cc:1722] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:39.889966 339 tablet_bootstrap.cc:654] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.904575 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 185c2bf7b9254947bb76b7fe0924559e with delete_type TABLET_DATA_DELETED (replaced by tablet 04fde7d1c1744c0597e43086ca4c9541) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.917047 32428 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:39.926487 1682 raft_consensus.cc:491] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:39.938071 1682 raft_consensus.cc:513] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.938740 1682 leader_election.cc:290] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:39.941313 32314 leader_election.cc:343] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: dc0dad440b494018bc0d6cfd40940bc1
W20250623 08:04:39.941690 32314 leader_election.cc:343] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: dc0dad440b494018bc0d6cfd40940bc1
I20250623 08:04:39.941781 32314 leader_election.cc:304] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:39.942124 1682 raft_consensus.cc:2747] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:39.952302 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 129361us
I20250623 08:04:39.952692 1789 consensus_queue.cc:1035] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:39.953961 32050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 9dbbdbebdf3b4b50bda31a94fc0de36f
I20250623 08:04:39.954551 32050 catalog_manager.cc:6652] ReplaceTablet: tablet 9dbbdbebdf3b4b50bda31a94fc0de36f deleted and replaced by tablet 79014324599840038cf53ca02cbc6f7e
I20250623 08:04:39.954877 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f with delete_type TABLET_DATA_DELETED (replaced by tablet 79014324599840038cf53ca02cbc6f7e) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.954932 32050 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_68d8428833794f57b663cdae5b7d6cb3" } modify_external_catalogs: true
I20250623 08:04:39.955142 32050 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_68d8428833794f57b663cdae5b7d6cb3" } modify_external_catalogs: true
I20250623 08:04:39.955262 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f with delete_type TABLET_DATA_DELETED (replaced by tablet 79014324599840038cf53ca02cbc6f7e) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.955735 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f with delete_type TABLET_DATA_DELETED (replaced by tablet 79014324599840038cf53ca02cbc6f7e) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.956894 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 30dfa986dfd44658a0ffb07796030520
I20250623 08:04:39.957397 32054 catalog_manager.cc:6652] ReplaceTablet: tablet 30dfa986dfd44658a0ffb07796030520 deleted and replaced by tablet 4a0615e6f8c346aba5e5f30362bc6de4
I20250623 08:04:39.957574 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 30dfa986dfd44658a0ffb07796030520 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a0615e6f8c346aba5e5f30362bc6de4) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:39.957733 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c1548afc4cff490a94e8a03d5fea9339" } new_table_name: "default.Table_2ab902dacb4a4c7898a6ab01df011eb4" modify_external_catalogs: true
I20250623 08:04:39.957883 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 30dfa986dfd44658a0ffb07796030520 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a0615e6f8c346aba5e5f30362bc6de4) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:39.959005 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 30dfa986dfd44658a0ffb07796030520 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a0615e6f8c346aba5e5f30362bc6de4) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:39.965517 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 57cd427592714247a2e075881f9827fb (DEFAULT_TABLE table=default.Table_f7811b4c393d498cbf95abe8a8cdcd38 [id=1c335fc6495f4256a2d7e1f4fd66e875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.965937 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57cd427592714247a2e075881f9827fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.966523 1245 ts_manager.cc:194] Registered new tserver with Master: 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:39.968348 1805 tablet.cc:1722] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:39.973567 1804 tablet.cc:1722] T 7419cb91360f49e1b3cb7e90732558c5 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:39.976547 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 57cd427592714247a2e075881f9827fb (DEFAULT_TABLE table=default.Table_f7811b4c393d498cbf95abe8a8cdcd38 [id=1c335fc6495f4256a2d7e1f4fd66e875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.976904 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57cd427592714247a2e075881f9827fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.977205 32627 tablet_service.cc:1468] Processing CreateTablet for tablet c1ab3afffac646139eea609b1a22787c (DEFAULT_TABLE table=default.Table_c4c658393f424eb1ac22133856d0d0b1 [id=44217c9c758e4530aa497a405168eac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.977619 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ab3afffac646139eea609b1a22787c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.978677 343 ts_tablet_manager.cc:1428] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.166s user 0.001s sys 0.001s
I20250623 08:04:39.978978 343 tablet_bootstrap.cc:492] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:39.979629 342 raft_consensus.cc:357] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.979836 342 raft_consensus.cc:383] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:39.979888 342 raft_consensus.cc:738] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:39.980098 342 consensus_queue.cc:260] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:39.980742 342 ts_tablet_manager.cc:1428] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.062s user 0.000s sys 0.003s
I20250623 08:04:39.981014 342 tablet_bootstrap.cc:492] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:39.981952 32353 tablet_service.cc:1468] Processing CreateTablet for tablet c1ab3afffac646139eea609b1a22787c (DEFAULT_TABLE table=default.Table_c4c658393f424eb1ac22133856d0d0b1 [id=44217c9c758e4530aa497a405168eac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.982292 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ab3afffac646139eea609b1a22787c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:39.982353 342 tablet_bootstrap.cc:654] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:39.983042 1810 tablet.cc:1722] T a1c82e4d2db64026a04c5c84bc1e09ec P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:39.984375 1809 tablet.cc:1722] T 31ecc74317d449e5b557b07a4fd14895 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250623 08:04:39.952692 32589 leader_election.cc:343] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:39.988561 32589 leader_election.cc:304] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:39.988991 1707 raft_consensus.cc:2747] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:39.989172 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 3cc8503b612d4eb7b8e42d9a3bdfd8f5 (DEFAULT_TABLE table=default.Table_1a60513fe9e94660aa640b5522394bbb [id=79c8424b812d4c98b8dbd8e1fdd5dd52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:39.989576 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cc8503b612d4eb7b8e42d9a3bdfd8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.000882 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 372a29156dab44f59d8ff435a51bdfcd (DEFAULT_TABLE table=default.Table_4903e809cc2d447fb94224ccbb05078d [id=0b8ff648531f42dd94998cc01b79bf3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.000969 32352 tablet_service.cc:1468] Processing CreateTablet for tablet a1cf375214c54f3491098db07faedd9f (DEFAULT_TABLE table=default.Table_56c6dd0a0ee64ed088889c23db0fd2e9 [id=4199d682d171458ab79e4a36807c7f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.001291 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 3cc8503b612d4eb7b8e42d9a3bdfd8f5 (DEFAULT_TABLE table=default.Table_1a60513fe9e94660aa640b5522394bbb [id=79c8424b812d4c98b8dbd8e1fdd5dd52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.001470 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cf375214c54f3491098db07faedd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.001811 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 372a29156dab44f59d8ff435a51bdfcd (DEFAULT_TABLE table=default.Table_4903e809cc2d447fb94224ccbb05078d [id=0b8ff648531f42dd94998cc01b79bf3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.002038 1807 tablet.cc:1722] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.002140 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 372a29156dab44f59d8ff435a51bdfcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.002142 1810 tablet.cc:1722] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.006871 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cc8503b612d4eb7b8e42d9a3bdfd8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.007349 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 372a29156dab44f59d8ff435a51bdfcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.008453 1813 tablet.cc:1722] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.009326 1245 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "59f7c614254f42009306b13337f06576" instance_seqno: 1750665860944300) as {username='slave'} at 127.31.69.2:54519; Asking this server to re-register.
I20250623 08:04:39.989010 32589 leader_election.cc:304] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:40.011420 32629 tablet_service.cc:1468] Processing CreateTablet for tablet a1cf375214c54f3491098db07faedd9f (DEFAULT_TABLE table=default.Table_56c6dd0a0ee64ed088889c23db0fd2e9 [id=4199d682d171458ab79e4a36807c7f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.011629 32568 heartbeater.cc:461] Registering TS with master...
W20250623 08:04:40.011672 32589 leader_election.cc:343] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.011775 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cf375214c54f3491098db07faedd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.012290 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38eecf4a94d4536b98d4f6e037a0094" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.012784 1707 raft_consensus.cc:2802] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:40.012908 1707 raft_consensus.cc:695] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:40.013181 1707 consensus_queue.cc:237] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.016076 342 tablet_bootstrap.cc:492] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:40.016204 342 ts_tablet_manager.cc:1397] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.003s
I20250623 08:04:40.016934 979 ts_tablet_manager.cc:1918] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:40.016902 342 raft_consensus.cc:357] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.017060 342 raft_consensus.cc:383] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.017107 979 log.cc:1199] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/f02f36d6c1174a23a1f1043a431d0b6b
I20250623 08:04:40.017114 342 raft_consensus.cc:738] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.017293 342 consensus_queue.cc:260] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.017777 342 ts_tablet_manager.cc:1428] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:40.017925 979 ts_tablet_manager.cc:1939] T f02f36d6c1174a23a1f1043a431d0b6b P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:40.018064 342 tablet_bootstrap.cc:492] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:40.018011 1814 tablet.cc:1722] T 7419cb91360f49e1b3cb7e90732558c5 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.018903 1806 raft_consensus.cc:491] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.019029 1806 raft_consensus.cc:513] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.019053 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 5b1a601763e44934a7db2f53d05e43a5 (DEFAULT_TABLE table=default.Table_0666030ad7ec4ef89335cdbaf0cca443 [id=bf6ca5aa8d5044868244a9b37cfa9519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.019358 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1a601763e44934a7db2f53d05e43a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.019603 1806 leader_election.cc:290] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.019979 32629 tablet_service.cc:1468] Processing CreateTablet for tablet a29c7fd219604bf39f40ccd09dedd2d2 (DEFAULT_TABLE table=default.Table_98f40c99d9ff4819b958cd84cc5dc3a9 [id=13cf327a200c490992163a70f5a275cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.020370 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29c7fd219604bf39f40ccd09dedd2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.020480 342 tablet_bootstrap.cc:654] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.022616 1810 tablet.cc:1722] T 31ecc74317d449e5b557b07a4fd14895 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.018808 32352 tablet_service.cc:1468] Processing CreateTablet for tablet a29c7fd219604bf39f40ccd09dedd2d2 (DEFAULT_TABLE table=default.Table_98f40c99d9ff4819b958cd84cc5dc3a9 [id=13cf327a200c490992163a70f5a275cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.023202 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 5b1a601763e44934a7db2f53d05e43a5 (DEFAULT_TABLE table=default.Table_0666030ad7ec4ef89335cdbaf0cca443 [id=bf6ca5aa8d5044868244a9b37cfa9519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.023383 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29c7fd219604bf39f40ccd09dedd2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.023394 343 tablet_bootstrap.cc:654] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.023561 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1a601763e44934a7db2f53d05e43a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.028375 1626 raft_consensus.cc:491] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.028384 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8a5e35c1974456b8079ab8907c2652e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.028510 1626 raft_consensus.cc:513] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.029104 1626 leader_election.cc:290] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.033140 1627 consensus_queue.cc:1035] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.033226 32453 leader_election.cc:304] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.033486 1627 raft_consensus.cc:3058] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.033566 32453 leader_election.cc:304] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.034065 1359 raft_consensus.cc:3058] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.036182 1627 raft_consensus.cc:2747] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.036527 1359 raft_consensus.cc:2747] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.038465 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 07aac1a69513468f80f40d295853279a (DEFAULT_TABLE table=default.Table_b08756509dd14eeeb598ef00e68600e2 [id=da8609b51169434c84a71ad27ca786b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.038719 32349 tablet_service.cc:1468] Processing CreateTablet for tablet 1a404aac062c41e8a06207d9d8685a4d (DEFAULT_TABLE table=default.Table_ac62bae678de4b11a971f7bbccd50303 [id=4f4663a5cad14a3593e16a52ec8aeece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.038916 1245 ts_manager.cc:194] Registered new tserver with Master: 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.038950 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07aac1a69513468f80f40d295853279a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.039417 32349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a404aac062c41e8a06207d9d8685a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.039430 1777 tablet.cc:1722] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.040535 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 07aac1a69513468f80f40d295853279a (DEFAULT_TABLE table=default.Table_b08756509dd14eeeb598ef00e68600e2 [id=da8609b51169434c84a71ad27ca786b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.040874 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07aac1a69513468f80f40d295853279a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.045740 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8a5e35c1974456b8079ab8907c2652e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.045956 32651 raft_consensus.cc:2466] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:40.045992 1812 tablet.cc:1722] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.047166 1245 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "125745849172449ba4be8244d61fdcb3" instance_seqno: 1750665861404587) as {username='slave'} at 127.31.69.3:50329; Asking this server to re-register.
I20250623 08:04:40.047379 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 081cef8b055341dbb98d6de3c918d2e2 (DEFAULT_TABLE table=default.Table_1911a70d3e224ae4b9805e345c99624b [id=9f504d8065a24602af6e2142069438a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.047736 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081cef8b055341dbb98d6de3c918d2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.048008 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7874a1d5cb9a464294ca32b2767efbf3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.048924 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.050843 1682 raft_consensus.cc:491] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.050966 1682 raft_consensus.cc:513] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:40.051198 1789 raft_consensus.cc:491] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.051328 1789 raft_consensus.cc:513] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.051504 1682 leader_election.cc:290] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.051981 1789 leader_election.cc:290] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.052438 339 tablet_bootstrap.cc:492] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:40.052886 339 ts_tablet_manager.cc:1397] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.407s user 0.006s sys 0.002s
I20250623 08:04:40.053638 339 raft_consensus.cc:357] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.053794 339 raft_consensus.cc:383] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.053856 339 raft_consensus.cc:738] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.054056 339 consensus_queue.cc:260] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.055120 339 ts_tablet_manager.cc:1428] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250623 08:04:40.055169 32314 leader_election.cc:343] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 79731106915d4cebb9dcc3c10386ffa9
W20250623 08:04:40.055415 32314 leader_election.cc:343] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 79731106915d4cebb9dcc3c10386ffa9
I20250623 08:04:40.055567 32314 leader_election.cc:304] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:40.055868 1682 raft_consensus.cc:2747] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.054064 1810 tablet.cc:1722] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:40.056981 32453 consensus_peers.cc:487] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 025365d2fa0a43dd983d395a05cc2af1. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:40.057308 32453 leader_election.cc:304] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.057677 32453 leader_election.cc:304] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.058360 1627 raft_consensus.cc:2802] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:40.058496 1627 raft_consensus.cc:491] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.058626 1627 raft_consensus.cc:3058] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.058743 1359 raft_consensus.cc:2802] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:40.058884 1359 raft_consensus.cc:695] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:40.059113 1359 consensus_queue.cc:237] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.058382 1806 raft_consensus.cc:491] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.060020 1806 raft_consensus.cc:513] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.060500 1806 leader_election.cc:290] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.061720 1627 raft_consensus.cc:513] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.062258 1627 leader_election.cc:290] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.063048 1771 raft_consensus.cc:491] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.063253 1771 raft_consensus.cc:513] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.063961 1771 leader_election.cc:290] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.064468 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 623e64e1cd7d4a458fb30ceece2989fd (DEFAULT_TABLE table=default.Table_8e6f4c63178440f4ba5c9f65d213df3d [id=3d71395c0fab47b68b1dcc47bfdc46fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.064844 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 623e64e1cd7d4a458fb30ceece2989fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.065557 32703 heartbeater.cc:461] Registering TS with master...
I20250623 08:04:40.066615 32069 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9752155c20ee42fbb7ff7b7d5b682444" candidate_term: 4 candidate_status { last_received { term: 3 index: 518 } } ignore_live_leader: false dest_uuid: "d31835ec4ad74332a00998d431492466" is_pre_election: true
I20250623 08:04:40.066968 343 tablet_bootstrap.cc:492] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:40.067044 32618 tablet_service.cc:1468] Processing CreateTablet for tablet c1b7896030ee4394bdabc785ab31b97e (DEFAULT_TABLE table=default.Table_fddbef2bc1d046689a8ef37cdcddcfa4 [id=e24b34aca192447a934b8775b709d8ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.067479 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b7896030ee4394bdabc785ab31b97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.068780 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306c4021b5d94a4996de07c222a8bfd8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.069670 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9f4829e103b460eb582beff3b966bb4" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.067152 343 ts_tablet_manager.cc:1397] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.000s
I20250623 08:04:40.071419 32345 tablet_service.cc:1468] Processing CreateTablet for tablet c1b7896030ee4394bdabc785ab31b97e (DEFAULT_TABLE table=default.Table_fddbef2bc1d046689a8ef37cdcddcfa4 [id=e24b34aca192447a934b8775b709d8ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.071462 32349 tablet_service.cc:1468] Processing CreateTablet for tablet 081cef8b055341dbb98d6de3c918d2e2 (DEFAULT_TABLE table=default.Table_1911a70d3e224ae4b9805e345c99624b [id=9f504d8065a24602af6e2142069438a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.071533 343 raft_consensus.cc:357] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.071818 343 raft_consensus.cc:383] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.071915 343 raft_consensus.cc:738] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.072160 343 consensus_queue.cc:260] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.072713 343 ts_tablet_manager.cc:1428] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:40.074016 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 623e64e1cd7d4a458fb30ceece2989fd (DEFAULT_TABLE table=default.Table_8e6f4c63178440f4ba5c9f65d213df3d [id=3d71395c0fab47b68b1dcc47bfdc46fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.077733 343 tablet_bootstrap.cc:492] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:40.081413 32453 consensus_peers.cc:487] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 724d662bcf754c74a8f3dff91ee258ae. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:40.081681 32453 consensus_peers.cc:487] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:40.081840 32453 leader_election.cc:304] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:40.082572 339 tablet_bootstrap.cc:492] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:40.082717 1806 raft_consensus.cc:2802] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:40.082821 1806 raft_consensus.cc:491] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.082902 1806 raft_consensus.cc:3058] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.082921 1789 raft_consensus.cc:491] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.083055 1789 raft_consensus.cc:513] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.083606 1789 leader_election.cc:290] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:40.083840 32589 leader_election.cc:343] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.085191 339 tablet_bootstrap.cc:654] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:40.089408 32589 leader_election.cc:343] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:40.090091 32589 leader_election.cc:343] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.090467 1707 consensus_queue.cc:1035] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.090171 343 tablet_bootstrap.cc:654] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.093004 979 tablet_replica.cc:331] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:40.093223 979 raft_consensus.cc:2241] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.093415 979 raft_consensus.cc:2270] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.094012 979 ts_tablet_manager.cc:1905] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.095906 1245 ts_manager.cc:194] Registered new tserver with Master: 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.096462 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 1a404aac062c41e8a06207d9d8685a4d (DEFAULT_TABLE table=default.Table_ac62bae678de4b11a971f7bbccd50303 [id=4f4663a5cad14a3593e16a52ec8aeece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.097009 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a404aac062c41e8a06207d9d8685a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.099107 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet f02f36d6c1174a23a1f1043a431d0b6b (table default.Table_75451c4863a34537a328617c53d46cae [id=be88c6ecf20d4ab1b3757c27ce998f87]) successfully deleted
I20250623 08:04:40.100525 1806 raft_consensus.cc:513] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.101195 456 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9752155c20ee42fbb7ff7b7d5b682444" candidate_term: 4 candidate_status { last_received { term: 3 index: 518 } } ignore_live_leader: false dest_uuid: "6644af0f4b2a4b998282b021ec1dc678" is_pre_election: true
I20250623 08:04:40.101263 981 ts_tablet_manager.cc:1918] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:40.101342 339 tablet_bootstrap.cc:492] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:40.101362 981 log.cc:1199] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/75a6ebe0d6b0463b8ddcb1a091a2cc6f
I20250623 08:04:40.101370 1806 leader_election.cc:290] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.101451 339 ts_tablet_manager.cc:1397] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20250623 08:04:40.101788 981 ts_tablet_manager.cc:1939] T 75a6ebe0d6b0463b8ddcb1a091a2cc6f P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:40.102203 339 raft_consensus.cc:357] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.102365 339 raft_consensus.cc:383] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.102447 339 raft_consensus.cc:738] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.102651 339 consensus_queue.cc:260] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.102909 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 190) took 5292 ms. Trace:
I20250623 08:04:40.103019 981 rpcz_store.cc:276] 0623 08:04:34.810503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.810545 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:39.868584 (+5058039us) ts_tablet_manager.cc:1145] Deleting tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f
0623 08:04:39.868593 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.032439 (+163846us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.101168 (+ 68729us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.102891 (+ 1723us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5057835}
I20250623 08:04:40.103143 339 ts_tablet_manager.cc:1428] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:40.103811 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 57cd427592714247a2e075881f9827fb (DEFAULT_TABLE table=default.Table_f7811b4c393d498cbf95abe8a8cdcd38 [id=1c335fc6495f4256a2d7e1f4fd66e875]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.104176 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57cd427592714247a2e075881f9827fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.105434 342 tablet_bootstrap.cc:492] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:40.105548 342 ts_tablet_manager.cc:1397] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.001s
I20250623 08:04:40.106194 342 raft_consensus.cc:357] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.106346 342 raft_consensus.cc:383] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.106421 342 raft_consensus.cc:738] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.106621 342 consensus_queue.cc:260] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.107093 32484 tablet_service.cc:1468] Processing CreateTablet for tablet c1ab3afffac646139eea609b1a22787c (DEFAULT_TABLE table=default.Table_c4c658393f424eb1ac22133856d0d0b1 [id=44217c9c758e4530aa497a405168eac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.107416 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1ab3afffac646139eea609b1a22787c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.109417 342 ts_tablet_manager.cc:1428] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:40.109655 342 tablet_bootstrap.cc:492] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:40.111006 342 tablet_bootstrap.cc:654] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.103518 339 tablet_bootstrap.cc:492] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:40.112748 339 tablet_bootstrap.cc:654] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.113171 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:40.113391 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:40.113801 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:40.114032 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:40.114413 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:40.114794 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:40.115056 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:40.115329 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:40.115522 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:40.115700 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:40.116262 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:40.116560 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet d5cd316e6f4447be958611ab07ed31fc (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:40.116787 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:40.116940 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:40.117599 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:40.117823 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_217651efa5dc4571b2a47cd3196c9e57 [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 1 got 0
I20250623 08:04:40.117981 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:40.118359 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:40.118892 343 tablet_bootstrap.cc:492] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:40.119058 343 ts_tablet_manager.cc:1397] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.001s
I20250623 08:04:40.119417 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 372a29156dab44f59d8ff435a51bdfcd (DEFAULT_TABLE table=default.Table_4903e809cc2d447fb94224ccbb05078d [id=0b8ff648531f42dd94998cc01b79bf3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.119612 32492 tablet_service.cc:1468] Processing CreateTablet for tablet a1cf375214c54f3491098db07faedd9f (DEFAULT_TABLE table=default.Table_56c6dd0a0ee64ed088889c23db0fd2e9 [id=4199d682d171458ab79e4a36807c7f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.119835 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 372a29156dab44f59d8ff435a51bdfcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.120083 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cf375214c54f3491098db07faedd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.119431 32484 tablet_service.cc:1468] Processing CreateTablet for tablet 3cc8503b612d4eb7b8e42d9a3bdfd8f5 (DEFAULT_TABLE table=default.Table_1a60513fe9e94660aa640b5522394bbb [id=79c8424b812d4c98b8dbd8e1fdd5dd52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.120407 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cc8503b612d4eb7b8e42d9a3bdfd8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.127544 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 5b1a601763e44934a7db2f53d05e43a5 (DEFAULT_TABLE table=default.Table_0666030ad7ec4ef89335cdbaf0cca443 [id=bf6ca5aa8d5044868244a9b37cfa9519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.127559 32492 tablet_service.cc:1468] Processing CreateTablet for tablet a29c7fd219604bf39f40ccd09dedd2d2 (DEFAULT_TABLE table=default.Table_98f40c99d9ff4819b958cd84cc5dc3a9 [id=13cf327a200c490992163a70f5a275cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.127928 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b1a601763e44934a7db2f53d05e43a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.130057 1213 leader_election.cc:304] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9752155c20ee42fbb7ff7b7d5b682444; no voters: 6644af0f4b2a4b998282b021ec1dc678, d31835ec4ad74332a00998d431492466
I20250623 08:04:40.130511 1781 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 08:04:40.132897 343 raft_consensus.cc:357] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.133168 343 raft_consensus.cc:383] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.133716 343 raft_consensus.cc:738] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.134068 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29c7fd219604bf39f40ccd09dedd2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.135003 343 consensus_queue.cc:260] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.135334 342 tablet_bootstrap.cc:492] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:40.135556 342 ts_tablet_manager.cc:1397] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250623 08:04:40.136006 343 ts_tablet_manager.cc:1428] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250623 08:04:40.136204 342 raft_consensus.cc:357] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.136363 342 raft_consensus.cc:383] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.136431 342 raft_consensus.cc:738] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.136610 342 consensus_queue.cc:260] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.137130 342 ts_tablet_manager.cc:1428] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250623 08:04:40.137434 342 tablet_bootstrap.cc:492] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:40.138715 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 48312.8us
I20250623 08:04:40.139007 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b7896030ee4394bdabc785ab31b97e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:40.139744 32589 leader_election.cc:343] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:40.140033 32589 leader_election.cc:343] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.140776 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "789d1c20aa72480ab71123c799bbde4c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.141556 342 tablet_bootstrap.cc:654] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:40.141867 32589 leader_election.cc:343] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.143051 32349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081cef8b055341dbb98d6de3c918d2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.144263 343 tablet_bootstrap.cc:492] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:40.145677 343 tablet_bootstrap.cc:654] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.147024 339 tablet_bootstrap.cc:492] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:40.147159 339 ts_tablet_manager.cc:1397] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20250623 08:04:40.147899 339 raft_consensus.cc:357] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.148061 339 raft_consensus.cc:383] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.148114 339 raft_consensus.cc:738] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.148298 339 consensus_queue.cc:260] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.148753 339 ts_tablet_manager.cc:1428] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:40.150269 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:40.150703 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:40.151158 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1c529e826fa84a919c018887620e4023 (table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]). Expected version 1 got 0
I20250623 08:04:40.151405 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:40.152050 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:40.152276 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:40.152981 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:40.153447 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:40.154259 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:40.154565 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:40.161842 339 tablet_bootstrap.cc:492] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:40.161842 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 623e64e1cd7d4a458fb30ceece2989fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.165421 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f (table default.Table_91b12dbe72c64d16a3334faf46445848 [id=d68afa3a431d4386b862bd4b47a5e1f0]) successfully deleted
I20250623 08:04:40.168538 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 191) took 5337 ms. Trace:
I20250623 08:04:40.168730 981 rpcz_store.cc:276] 0623 08:04:34.831126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.831198 (+ 72us) service_pool.cc:225] Handling call
0623 08:04:40.152835 (+5321637us) ts_tablet_manager.cc:1145] Deleting tablet d4221d7500d1458db75926d7b274a654
0623 08:04:40.168452 (+ 15617us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.168520 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5254121}
I20250623 08:04:40.169096 981 tablet_replica.cc:331] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:40.169335 981 raft_consensus.cc:2241] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.170207 981 raft_consensus.cc:2270] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.171165 981 ts_tablet_manager.cc:1905] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.179895 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 07aac1a69513468f80f40d295853279a (DEFAULT_TABLE table=default.Table_b08756509dd14eeeb598ef00e68600e2 [id=da8609b51169434c84a71ad27ca786b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.180307 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07aac1a69513468f80f40d295853279a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.171660 339 tablet_bootstrap.cc:654] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.185704 979 ts_tablet_manager.cc:1918] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:40.190228 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 081cef8b055341dbb98d6de3c918d2e2 (DEFAULT_TABLE table=default.Table_1911a70d3e224ae4b9805e345c99624b [id=9f504d8065a24602af6e2142069438a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.190799 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081cef8b055341dbb98d6de3c918d2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.195037 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 1a404aac062c41e8a06207d9d8685a4d (DEFAULT_TABLE table=default.Table_ac62bae678de4b11a971f7bbccd50303 [id=4f4663a5cad14a3593e16a52ec8aeece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.195297 1682 raft_consensus.cc:491] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.195425 1682 raft_consensus.cc:513] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.196002 1682 leader_election.cc:290] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.196506 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a404aac062c41e8a06207d9d8685a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.193274 343 tablet_bootstrap.cc:492] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:40.198984 343 ts_tablet_manager.cc:1397] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.001s
I20250623 08:04:40.199393 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736eb56dae194df2af90bf33d3294a4e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.199554 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736eb56dae194df2af90bf33d3294a4e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.199769 32514 raft_consensus.cc:2391] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:40.199875 343 raft_consensus.cc:357] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.200111 343 raft_consensus.cc:383] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.200181 343 raft_consensus.cc:738] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.200439 32314 leader_election.cc:304] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:40.200492 343 consensus_queue.cc:260] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.200723 1682 raft_consensus.cc:3058] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.201277 343 ts_tablet_manager.cc:1428] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:40.201413 979 log.cc:1199] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/f4a96ae9190640dea11bd11a9f0e4372
I20250623 08:04:40.201719 979 ts_tablet_manager.cc:1939] T f4a96ae9190640dea11bd11a9f0e4372 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:40.202781 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:40.203163 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a9ea74d20c5f4eefbcb8508ed1774026 (table default.Table_0b9c0d17ddcd422eb8035e69884f3da6 [id=ced0b4cf62924dbfadad02109349bea0]). Expected version 1 got 0
I20250623 08:04:40.203564 1682 raft_consensus.cc:2747] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.204559 343 tablet_bootstrap.cc:492] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:40.205144 1682 consensus_queue.cc:1035] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.205914 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7419cb91360f49e1b3cb7e90732558c5 (table default.Table_161ad5ddd1cb40e0989e5c237a205db1 [id=7cc0919082d04fc4a02aa856a42e5529]). Expected version 1 got 0
I20250623 08:04:40.206560 343 tablet_bootstrap.cc:654] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.207226 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:40.207856 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:40.208242 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:40.208495 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:40.208681 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:40.208806 339 tablet_bootstrap.cc:492] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:40.208927 339 ts_tablet_manager.cc:1397] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.001s
I20250623 08:04:40.209007 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:40.209565 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:40.209626 339 raft_consensus.cc:357] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.209780 339 raft_consensus.cc:383] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.209796 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:40.209831 339 raft_consensus.cc:738] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.210011 339 consensus_queue.cc:260] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.210417 32055 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 5 got 0
I20250623 08:04:40.210456 339 ts_tablet_manager.cc:1428] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250623 08:04:40.211298 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 320) took 1079 ms. Trace:
I20250623 08:04:40.211406 979 rpcz_store.cc:276] 0623 08:04:39.131972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.132027 (+ 55us) service_pool.cc:225] Handling call
0623 08:04:40.092956 (+960929us) ts_tablet_manager.cc:1145] Deleting tablet f4a96ae9190640dea11bd11a9f0e4372
0623 08:04:40.092966 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.097652 (+ 4686us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.185604 (+ 87952us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.211277 (+ 25673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":960665}
I20250623 08:04:40.212232 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 317) took 1082 ms. Trace:
I20250623 08:04:40.212373 979 rpcz_store.cc:276] 0623 08:04:39.130077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.132448 (+ 2371us) service_pool.cc:225] Handling call
0623 08:04:40.211869 (+1079421us) ts_tablet_manager.cc:1145] Deleting tablet f0e24d8d890d4d1db051f172b3ec96ec
0623 08:04:40.212048 (+ 179us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.212219 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1079232}
I20250623 08:04:40.213059 979 tablet_replica.cc:331] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:40.213285 979 raft_consensus.cc:2241] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.213521 979 raft_consensus.cc:2270] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.214104 979 ts_tablet_manager.cc:1905] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.214269 339 tablet_bootstrap.cc:492] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:40.220932 339 tablet_bootstrap.cc:654] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.229202 1707 raft_consensus.cc:491] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.229354 1707 raft_consensus.cc:513] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.230024 1707 leader_election.cc:290] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.230738 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b5233522f43c7aed3c9770adb5c03" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:40.231266 32589 leader_election.cc:343] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.232383 981 ts_tablet_manager.cc:1918] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:40.232589 981 log.cc:1199] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/6c6f98e2db1a4b0789c5819b2c7d4bb6
I20250623 08:04:40.233029 981 ts_tablet_manager.cc:1939] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:40.234066 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 206) took 5166 ms. Trace:
I20250623 08:04:40.234272 981 rpcz_store.cc:276] 0623 08:04:35.067145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.081067 (+ 13922us) service_pool.cc:225] Handling call
0623 08:04:40.169057 (+5087990us) ts_tablet_manager.cc:1145] Deleting tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6
0623 08:04:40.169066 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.195093 (+ 26027us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.232245 (+ 37152us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.234040 (+ 1795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5034834}
I20250623 08:04:40.235100 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 207) took 5048 ms. Trace:
I20250623 08:04:40.235235 981 rpcz_store.cc:276] 0623 08:04:35.187242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.187295 (+ 53us) service_pool.cc:225] Handling call
0623 08:04:40.234976 (+5047681us) ts_tablet_manager.cc:1145] Deleting tablet 36c8714cfbd84dd38cf5c0f305ef6843
0623 08:04:40.234985 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.235084 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5047393}
I20250623 08:04:40.235718 981 tablet_replica.cc:331] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:40.235982 981 raft_consensus.cc:2241] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.236171 981 raft_consensus.cc:2270] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.236690 981 ts_tablet_manager.cc:1905] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.239971 343 tablet_bootstrap.cc:492] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:40.240147 343 ts_tablet_manager.cc:1397] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.005s
I20250623 08:04:40.240823 343 raft_consensus.cc:357] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.241039 343 raft_consensus.cc:383] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.241122 343 raft_consensus.cc:738] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.241361 343 consensus_queue.cc:260] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.241979 343 ts_tablet_manager.cc:1428] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:40.242400 343 tablet_bootstrap.cc:492] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:40.243307 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet f4a96ae9190640dea11bd11a9f0e4372 (table default.Table_0666030ad7ec4ef89335cdbaf0cca443 [id=bf6ca5aa8d5044868244a9b37cfa9519]) successfully deleted
W20250623 08:04:40.243724 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f0e24d8d890d4d1db051f172b3ec96ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e24d8d890d4d1db051f172b3ec96ec already in progress: creating tablet
W20250623 08:04:40.244282 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet d4221d7500d1458db75926d7b274a654 because the tablet was not found. No further retry: Not found: Tablet not found: d4221d7500d1458db75926d7b274a654
I20250623 08:04:40.244515 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]) successfully deleted
W20250623 08:04:40.244758 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 36c8714cfbd84dd38cf5c0f305ef6843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36c8714cfbd84dd38cf5c0f305ef6843 already in progress: creating tablet
I20250623 08:04:40.245558 32484 tablet_service.cc:1468] Processing CreateTablet for tablet c1b7896030ee4394bdabc785ab31b97e (DEFAULT_TABLE table=default.Table_fddbef2bc1d046689a8ef37cdcddcfa4 [id=e24b34aca192447a934b8775b709d8ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.246073 343 tablet_bootstrap.cc:654] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.246112 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b7896030ee4394bdabc785ab31b97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.252403 979 ts_tablet_manager.cc:1918] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250623 08:04:40.252532 979 log.cc:1199] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/ba3d2fa01d574dd992dc904ac024ba3a
I20250623 08:04:40.248369 339 tablet_bootstrap.cc:492] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:40.252696 339 ts_tablet_manager.cc:1397] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250623 08:04:40.252975 979 ts_tablet_manager.cc:1939] T ba3d2fa01d574dd992dc904ac024ba3a P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:40.253540 339 raft_consensus.cc:357] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.253712 339 raft_consensus.cc:383] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.253739 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 321) took 1056 ms. Trace:
I20250623 08:04:40.253778 339 raft_consensus.cc:738] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.253827 979 rpcz_store.cc:276] 0623 08:04:39.197223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.197328 (+ 105us) service_pool.cc:225] Handling call
0623 08:04:40.212804 (+1015476us) ts_tablet_manager.cc:1145] Deleting tablet ba3d2fa01d574dd992dc904ac024ba3a
0623 08:04:40.213017 (+ 213us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.232700 (+ 19683us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.252298 (+ 19598us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.253717 (+ 1419us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1015255}
I20250623 08:04:40.253979 339 consensus_queue.cc:260] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.254459 339 ts_tablet_manager.cc:1428] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:40.254735 339 tablet_bootstrap.cc:492] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:40.256299 339 tablet_bootstrap.cc:654] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.258669 32484 tablet_service.cc:1468] Processing CreateTablet for tablet 623e64e1cd7d4a458fb30ceece2989fd (DEFAULT_TABLE table=default.Table_8e6f4c63178440f4ba5c9f65d213df3d [id=3d71395c0fab47b68b1dcc47bfdc46fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.266690 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 323) took 1043 ms. Trace:
I20250623 08:04:40.266875 979 rpcz_store.cc:276] 0623 08:04:39.223578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.223768 (+ 190us) service_pool.cc:225] Handling call
0623 08:04:40.266545 (+1042777us) ts_tablet_manager.cc:1145] Deleting tablet d6ed4cb252b1452087f8ee53ec78d985
0623 08:04:40.266560 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.266666 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042568}
I20250623 08:04:40.267472 979 tablet_replica.cc:331] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:40.267721 979 raft_consensus.cc:2241] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.265877 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 623e64e1cd7d4a458fb30ceece2989fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.268256 979 raft_consensus.cc:2270] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.268802 979 ts_tablet_manager.cc:1905] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.275441 981 ts_tablet_manager.cc:1918] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:40.275960 343 tablet_bootstrap.cc:492] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:40.276105 343 ts_tablet_manager.cc:1397] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.001s
I20250623 08:04:40.277171 952 ts_tablet_manager.cc:1918] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:40.282595 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet ba3d2fa01d574dd992dc904ac024ba3a (table default.Table_0b9112de297847f889ab29b68f4dfb8c [id=6c57db68f5ed45a3b64d1e3a02ce8a2a]) successfully deleted
I20250623 08:04:40.289084 343 raft_consensus.cc:357] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.289296 343 raft_consensus.cc:383] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.289343 343 raft_consensus.cc:738] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.289568 343 consensus_queue.cc:260] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.296121 343 ts_tablet_manager.cc:1428] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250623 08:04:40.296571 343 tablet_bootstrap.cc:492] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:40.296967 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d6ed4cb252b1452087f8ee53ec78d985 because the tablet was not found. No further retry: Not found: Tablet not found: d6ed4cb252b1452087f8ee53ec78d985
I20250623 08:04:40.305971 343 tablet_bootstrap.cc:654] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.278730 1682 raft_consensus.cc:491] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.291464 1303 consensus_queue.cc:1035] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.309504 1682 raft_consensus.cc:513] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.310369 1682 leader_election.cc:290] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.312114 981 log.cc:1199] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/79731106915d4cebb9dcc3c10386ffa9
I20250623 08:04:40.312999 979 ts_tablet_manager.cc:1918] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:40.314325 981 ts_tablet_manager.cc:1939] T 79731106915d4cebb9dcc3c10386ffa9 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:40.314563 979 log.cc:1199] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/d0b2af698b6e4ed0ae66a9bb32be7b31
I20250623 08:04:40.315418 952 log.cc:1199] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/6c6f98e2db1a4b0789c5819b2c7d4bb6
I20250623 08:04:40.315893 979 ts_tablet_manager.cc:1939] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:40.315965 952 ts_tablet_manager.cc:1939] T 6c6f98e2db1a4b0789c5819b2c7d4bb6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:40.316387 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87648b11b10b45fc9cc8c998bbceae92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.316660 32649 raft_consensus.cc:2391] T 87648b11b10b45fc9cc8c998bbceae92 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:40.316879 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 214) took 4961 ms. Trace:
I20250623 08:04:40.317181 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 176) took 5434 ms. Trace:
I20250623 08:04:40.317214 981 rpcz_store.cc:276] 0623 08:04:35.355746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.366824 (+ 11078us) service_pool.cc:225] Handling call
0623 08:04:40.235678 (+4868854us) ts_tablet_manager.cc:1145] Deleting tablet 79731106915d4cebb9dcc3c10386ffa9
0623 08:04:40.235687 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.267830 (+ 32143us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.275318 (+ 7488us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.316856 (+ 41538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4868572}
I20250623 08:04:40.317341 952 rpcz_store.cc:276] 0623 08:04:34.883077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.894149 (+ 11072us) service_pool.cc:225] Handling call
0623 08:04:39.844959 (+4950810us) ts_tablet_manager.cc:1145] Deleting tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6
0623 08:04:39.844968 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.177852 (+332884us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.277042 (+ 99190us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.317163 (+ 40121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4945063}
I20250623 08:04:40.306560 342 tablet_bootstrap.cc:492] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:40.317945 342 ts_tablet_manager.cc:1397] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.181s user 0.044s sys 0.044s
I20250623 08:04:40.318004 952 tablet_replica.cc:331] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:40.318286 952 raft_consensus.cc:2241] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.318444 1682 consensus_queue.cc:1035] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.318603 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 79731106915d4cebb9dcc3c10386ffa9 (table default.Table_d44a2132435f43e7847bfa349645a605 [id=b42459e9ca5e475cb5083d68bb6349be]) successfully deleted
I20250623 08:04:40.318732 342 raft_consensus.cc:357] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.318920 342 raft_consensus.cc:383] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.318969 342 raft_consensus.cc:738] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.319154 342 consensus_queue.cc:260] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.311941 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87648b11b10b45fc9cc8c998bbceae92" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.319597 342 ts_tablet_manager.cc:1428] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250623 08:04:40.278592 32453 leader_election.cc:343] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.320870 1809 tablet.cc:1717] T cdc9d47f2232403fa71e14a6fe6f33c3 P 59f7c614254f42009306b13337f06576: Skipping requested alter to schema version 2, tablet already version 2
I20250623 08:04:40.320642 32314 leader_election.cc:304] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:40.321794 1682 raft_consensus.cc:3058] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.322255 342 tablet_bootstrap.cc:492] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:40.324689 1682 raft_consensus.cc:2747] T 87648b11b10b45fc9cc8c998bbceae92 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.325181 342 tablet_bootstrap.cc:654] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.325465 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36c8714cfbd84dd38cf5c0f305ef6843" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.325634 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a96ae9190640dea11bd11a9f0e4372" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.326344 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42c861e29e1e4e549021dae7255a856c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.326467 1806 raft_consensus.cc:491] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.326624 1806 raft_consensus.cc:513] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.327347 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8a5e35c1974456b8079ab8907c2652e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.327358 1806 leader_election.cc:290] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.328380 32379 raft_consensus.cc:3058] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.333020 1810 tablet.cc:1717] T cdc9d47f2232403fa71e14a6fe6f33c3 P 7351aa31800245659edc9837e27daffd: Skipping requested alter to schema version 2, tablet already version 2
I20250623 08:04:40.334091 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.334270 32517 raft_consensus.cc:2466] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:40.334673 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7874a1d5cb9a464294ca32b2767efbf3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:40.335120 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9f4829e103b460eb582beff3b966bb4" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.336788 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]) successfully deleted
I20250623 08:04:40.340265 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36c8714cfbd84dd38cf5c0f305ef6843" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.340463 32651 raft_consensus.cc:2466] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:40.340792 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 326) took 1115 ms. Trace:
I20250623 08:04:40.340868 979 rpcz_store.cc:276] 0623 08:04:39.225184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.225784 (+ 600us) service_pool.cc:225] Handling call
0623 08:04:40.267418 (+1041634us) ts_tablet_manager.cc:1145] Deleting tablet d0b2af698b6e4ed0ae66a9bb32be7b31
0623 08:04:40.267429 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.272114 (+ 4685us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.312886 (+ 40772us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.340780 (+ 27894us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041483}
I20250623 08:04:40.348258 1804 tablet.cc:1722] T 31ecc74317d449e5b557b07a4fd14895 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.348879 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42c861e29e1e4e549021dae7255a856c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:40.349048 32648 raft_consensus.cc:3058] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.351925 32648 raft_consensus.cc:2466] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:40.353739 32519 raft_consensus.cc:2466] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:40.354038 1805 tablet.cc:1722] T 6c33fa0a7ff34d2985c5e07fcf88eaa1 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.354116 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8a5e35c1974456b8079ab8907c2652e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:40.354300 32647 raft_consensus.cc:3058] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.354363 1851 tablet.cc:1722] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.357753 32647 raft_consensus.cc:2466] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:40.362324 1805 tablet.cc:1722] T a1c82e4d2db64026a04c5c84bc1e09ec P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.365854 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 327) took 1140 ms. Trace:
I20250623 08:04:40.365962 979 rpcz_store.cc:276] 0623 08:04:39.225270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.225969 (+ 699us) service_pool.cc:225] Handling call
0623 08:04:40.341240 (+1115271us) ts_tablet_manager.cc:1145] Deleting tablet a414144276f6484ea4690204752e2142
0623 08:04:40.341250 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.365831 (+ 24581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1115158}
I20250623 08:04:40.366456 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38eecf4a94d4536b98d4f6e037a0094" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:40.366937 979 tablet_replica.cc:331] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:40.367174 979 raft_consensus.cc:2241] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.367343 979 raft_consensus.cc:2270] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.367888 979 ts_tablet_manager.cc:1905] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.370012 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet d0b2af698b6e4ed0ae66a9bb32be7b31 (table default.Table_d3f72251faee462c9c301b8aff464bb5 [id=b7671dfa27be46bb8b811b12016434e2]) successfully deleted
W20250623 08:04:40.370374 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet a414144276f6484ea4690204752e2142 because the tablet was not found. No further retry: Not found: Tablet not found: a414144276f6484ea4690204752e2142
I20250623 08:04:40.371145 952 raft_consensus.cc:2270] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.371860 952 ts_tablet_manager.cc:1905] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.375101 32514 raft_consensus.cc:2466] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:40.376677 339 tablet_bootstrap.cc:492] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:40.377785 339 ts_tablet_manager.cc:1397] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.123s user 0.004s sys 0.004s
I20250623 08:04:40.379137 339 raft_consensus.cc:357] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.380520 339 raft_consensus.cc:383] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.380645 339 raft_consensus.cc:738] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.380946 339 consensus_queue.cc:260] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.381630 339 ts_tablet_manager.cc:1428] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:40.381968 339 tablet_bootstrap.cc:492] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:40.384135 339 tablet_bootstrap.cc:654] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.388614 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b5233522f43c7aed3c9770adb5c03" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.389104 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.389290 32646 raft_consensus.cc:2466] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:40.389204 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306c4021b5d94a4996de07c222a8bfd8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.389760 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "789d1c20aa72480ab71123c799bbde4c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.390064 32514 raft_consensus.cc:2466] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:40.384104 1682 raft_consensus.cc:491] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.404868 1682 raft_consensus.cc:513] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.405709 1682 leader_election.cc:290] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.386253 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 213) took 5030 ms. Trace:
I20250623 08:04:40.405989 981 rpcz_store.cc:276] 0623 08:04:35.355400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.355447 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:40.386138 (+5030691us) ts_tablet_manager.cc:1145] Deleting tablet 32c82141edea4068b54755a85c6d7fb8
0623 08:04:40.386149 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.386231 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5008460}
I20250623 08:04:40.406651 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 212) took 5051 ms. Trace:
I20250623 08:04:40.406844 981 rpcz_store.cc:276] 0623 08:04:35.355272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.355792 (+ 520us) service_pool.cc:225] Handling call
0623 08:04:40.406415 (+5050623us) ts_tablet_manager.cc:1145] Deleting tablet b980c746a3c042bfae90aa076acbb1df
0623 08:04:40.406530 (+ 115us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.406625 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5028653}
I20250623 08:04:40.407363 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 215) took 4890 ms. Trace:
I20250623 08:04:40.407501 981 rpcz_store.cc:276] 0623 08:04:35.517337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.517399 (+ 62us) service_pool.cc:225] Handling call
0623 08:04:40.407228 (+4889829us) ts_tablet_manager.cc:1145] Deleting tablet 501b5233522f43c7aed3c9770adb5c03
0623 08:04:40.407240 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.407346 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4889598}
I20250623 08:04:40.407889 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 216) took 4890 ms. Trace:
I20250623 08:04:40.408001 981 rpcz_store.cc:276] 0623 08:04:35.517807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.517852 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:40.407804 (+4889952us) ts_tablet_manager.cc:1145] Deleting tablet 789d1c20aa72480ab71123c799bbde4c
0623 08:04:40.407811 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.407875 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4889789}
I20250623 08:04:40.408398 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 217) took 4860 ms. Trace:
I20250623 08:04:40.408514 981 rpcz_store.cc:276] 0623 08:04:35.548408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.548462 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:40.408260 (+4859798us) ts_tablet_manager.cc:1145] Deleting tablet db81250d1e164a7d94affd764fc0ff0c
0623 08:04:40.408269 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.408386 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4836981}
I20250623 08:04:40.408867 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 220) took 4413 ms. Trace:
I20250623 08:04:40.408984 981 rpcz_store.cc:276] 0623 08:04:35.995026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.995079 (+ 53us) service_pool.cc:225] Handling call
0623 08:04:40.408776 (+4413697us) ts_tablet_manager.cc:1145] Deleting tablet d19d468ce13d4108a9badc36fc503306
0623 08:04:40.408784 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.408854 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4405876}
I20250623 08:04:40.409262 981 tablet_replica.cc:331] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
W20250623 08:04:40.393664 32314 consensus_peers.cc:487] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9f8aa424bbb4aa8a907259270d46656. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:40.410845 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:40.413271 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 32c82141edea4068b54755a85c6d7fb8 because the tablet was not found. No further retry: Not found: Tablet not found: 32c82141edea4068b54755a85c6d7fb8
W20250623 08:04:40.413609 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet b980c746a3c042bfae90aa076acbb1df because the tablet was not found. No further retry: Not found: Tablet not found: b980c746a3c042bfae90aa076acbb1df
W20250623 08:04:40.413641 32314 leader_election.cc:343] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 18ea63263b9b466e922169eecafab747
W20250623 08:04:40.413808 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 501b5233522f43c7aed3c9770adb5c03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 501b5233522f43c7aed3c9770adb5c03 already in progress: creating tablet
W20250623 08:04:40.413978 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 789d1c20aa72480ab71123c799bbde4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 789d1c20aa72480ab71123c799bbde4c already in progress: creating tablet
W20250623 08:04:40.414201 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet db81250d1e164a7d94affd764fc0ff0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db81250d1e164a7d94affd764fc0ff0c already in progress: creating tablet
I20250623 08:04:40.414232 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ea63263b9b466e922169eecafab747" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:40.414427 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet d19d468ce13d4108a9badc36fc503306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d19d468ce13d4108a9badc36fc503306 already in progress: creating tablet
I20250623 08:04:40.414610 32514 raft_consensus.cc:2466] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:40.415266 32314 leader_election.cc:304] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:40.416740 1682 raft_consensus.cc:2802] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:40.416874 1682 raft_consensus.cc:491] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.416987 1682 raft_consensus.cc:3058] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.419981 1682 raft_consensus.cc:513] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.420655 1682 leader_election.cc:290] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.421005 981 raft_consensus.cc:2241] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.421205 981 raft_consensus.cc:2270] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.421813 981 ts_tablet_manager.cc:1905] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.422240 342 tablet_bootstrap.cc:492] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:40.422585 342 ts_tablet_manager.cc:1397] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.100s user 0.016s sys 0.025s
I20250623 08:04:40.423105 32589 leader_election.cc:304] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:40.423609 32589 leader_election.cc:343] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.423712 32589 leader_election.cc:304] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.423678 32453 leader_election.cc:304] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:40.423856 32453 raft_consensus.cc:2684] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:40.424093 1359 consensus_queue.cc:1035] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.424201 1789 raft_consensus.cc:2802] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250623 08:04:40.424180 32453 leader_election.cc:343] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: f4a96ae9190640dea11bd11a9f0e4372
I20250623 08:04:40.424268 1789 raft_consensus.cc:491] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.424367 1789 raft_consensus.cc:3058] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.424437 32453 leader_election.cc:304] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:40.424671 1359 raft_consensus.cc:2802] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:40.424799 1359 raft_consensus.cc:695] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:40.425036 1359 consensus_queue.cc:237] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:40.425282 32453 consensus_peers.cc:487] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bfd3e263464c4f2c8af9262acf211193. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:40.425417 32453 consensus_peers.cc:487] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0b2af698b6e4ed0ae66a9bb32be7b31. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:40.426493 32453 leader_election.cc:304] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:40.426717 1507 raft_consensus.cc:2802] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:40.426855 1507 raft_consensus.cc:695] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:40.426977 1707 raft_consensus.cc:2747] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.427055 1507 consensus_queue.cc:237] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.427884 1789 raft_consensus.cc:513] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.428208 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18ea63263b9b466e922169eecafab747" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:40.428375 32516 raft_consensus.cc:3058] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.428603 1789 leader_election.cc:290] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.431170 32516 raft_consensus.cc:2466] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:40.445401 343 tablet_bootstrap.cc:492] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:40.445652 343 ts_tablet_manager.cc:1397] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.149s user 0.009s sys 0.002s
I20250623 08:04:40.447633 979 ts_tablet_manager.cc:1918] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:40.447858 979 log.cc:1199] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/52d0a356c74f4daea9093c998b4bf73e
I20250623 08:04:40.448280 979 ts_tablet_manager.cc:1939] T 52d0a356c74f4daea9093c998b4bf73e P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:40.449141 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 328) took 1223 ms. Trace:
I20250623 08:04:40.449318 979 rpcz_store.cc:276] 0623 08:04:39.225355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.226104 (+ 749us) service_pool.cc:225] Handling call
0623 08:04:40.366868 (+1140764us) ts_tablet_manager.cc:1145] Deleting tablet 52d0a356c74f4daea9093c998b4bf73e
0623 08:04:40.366882 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.401324 (+ 34442us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.404924 (+ 3600us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.449123 (+ 44199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1140634}
I20250623 08:04:40.449632 1659 raft_consensus.cc:491] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.449743 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:40.449774 1659 raft_consensus.cc:513] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.450119 979 tablet_replica.cc:331] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:40.450294 979 raft_consensus.cc:2241] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.450338 1659 leader_election.cc:290] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.450425 979 raft_consensus.cc:2270] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.450950 979 ts_tablet_manager.cc:1905] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.451418 32453 leader_election.cc:304] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:40.451916 1507 raft_consensus.cc:2802] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:40.452004 1507 raft_consensus.cc:491] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.452144 1507 raft_consensus.cc:3058] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:39.816593 1267 rpcz_store.cc:276] 0623 08:04:34.848469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:34.848532 (+ 63us) service_pool.cc:225] Handling call
0623 08:04:39.215933 (+4367401us) raft_consensus.cc:1469] Updating replica for 162 ops
0623 08:04:39.239226 (+ 23293us) raft_consensus.cc:1532] Early marking committed up to index 356
0623 08:04:39.244663 (+ 5437us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.244669 (+ 6us) write_op.cc:454] Released schema lock
0623 08:04:39.304255 (+ 59586us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.304263 (+ 8us) write_op.cc:454] Released schema lock
0623 08:04:39.314984 (+ 10721us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.314990 (+ 6us) write_op.cc:454] Released schema lock
0623 08:04:39.315150 (+ 160us) raft_consensus.cc:1537] Triggering prepare for 162 ops
0623 08:04:39.316810 (+ 1660us) write_op.cc:270] Start()
0623 08:04:39.316853 (+ 43us) write_op.cc:276] Timestamp: P: 1750665869959929 usec, L: 0
0623 08:04:39.317697 (+ 844us) write_op.cc:270] Start()
0623 08:04:39.317723 (+ 26us) write_op.cc:276] Timestamp: P: 1750665869960724 usec, L: 0
0623 08:04:39.318245 (+ 522us) write_op.cc:270] Start()
0623 08:04:39.318267 (+ 22us) write_op.cc:276] Timestamp: P: 1750665869961296 usec, L: 0
0623 08:04:39.318735 (+ 468us) write_op.cc:270] Start()
0623 08:04:39.318808 (+ 73us) write_op.cc:276] Timestamp: P: 1750665869962059 usec, L: 0
0623 08:04:39.319284 (+ 476us) write_op.cc:270] Start()
0623 08:04:39.319304 (+ 20us) write_op.cc:276] Timestamp: P: 1750665869993377 usec, L: 0
0623 08:04:39.319746 (+ 442us) write_op.cc:270] Start()
0623 08:04:39.319766 (+ 20us) write_op.cc:276] Timestamp: P: 1750665869994366 usec, L: 0
0623 08:04:39.320178 (+ 412us) write_op.cc:270] Start()
0623 08:04:39.320198 (+ 20us) write_op.cc:276] Timestamp: P: 1750665870133464 usec, L: 0
0623 08:04:39.320640 (+ 442us) write_op.cc:270] Start()
0623 08:04:39.320661 (+ 21us) write_op.cc:276] Timestamp: P: 1750665870200295 usec, L: 0
0623 08:04:39.321088 (+ 427us) write_op.cc:270] Start()
0623 08:04:39.321120 (+ 32us) write_op.cc:276] Timestamp: P: 1750665870201261 usec, L: 0
0623 08:04:39.322475 (+ 1355us) write_op.cc:270] Start()
0623 08:04:39.322500 (+ 25us) write_op.cc:276] Timestamp: P: 1750665870212806 usec, L: 0
0623 08:04:39.323110 (+ 610us) write_op.cc:270] Start()
0623 08:04:39.323133 (+ 23us) write_op.cc:276] Timestamp: P: 1750665870234422 usec, L: 0
0623 08:04:39.323609 (+ 476us) write_op.cc:270] Start()
0623 08:04:39.323641 (+ 32us) write_op.cc:276] Timestamp: P: 1750665870289547 usec, L: 0
0623 08:04:39.337424 (+ 13783us) write_op.cc:270] Start()
0623 08:04:39.337465 (+ 41us) write_op.cc:276] Timestamp: P: 1750665870305686 usec, L: 0
0623 08:04:39.337651 (+ 186us) write_op.cc:270] Start()
0623 08:04:39.337680 (+ 29us) write_op.cc:276] Timestamp: P: 1750665870312102 usec, L: 0
0623 08:04:39.337815 (+ 135us) write_op.cc:270] Start()
0623 08:04:39.337835 (+ 20us) write_op.cc:276] Timestamp: P: 1750665870314434 usec, L: 0
0623 08:04:39.339397 (+ 1562us) write_op.cc:270] Start()
0623 08:04:39.339425 (+ 28us) write_op.cc:276] Timestamp: P: 1750665870339571 usec, L: 0
0623 08:04:39.339570 (+ 145us) write_op.cc:270] Start()
0623 08:04:39.339598 (+ 28us) write_op.cc:276] Timestamp: P: 1750665870466585 usec, L: 0
0623 08:04:39.339730 (+ 132us) write_op.cc:270] Start()
0623 08:04:39.339748 (+ 18us) write_op.cc:276] Timestamp: P: 1750665870477786 usec, L: 0
0623 08:04:39.339882 (+ 134us) write_op.cc:270] Start()
0623 08:04:39.339901 (+ 19us) write_op.cc:276] Timestamp: P: 1750665870480061 usec, L: 0
0623 08:04:39.340032 (+ 131us) write_op.cc:270] Start()
0623 08:04:39.340052 (+ 20us) write_op.cc:276] Timestamp: P: 1750665870512507 usec, L: 0
0623 08:04:39.340186 (+ 134us) write_op.cc:270] Start()
0623 08:04:39.340239 (+ 53us) write_op.cc:276] Timestamp: P: 1750665870522038 usec, L: 0
0623 08:04:39.340375 (+ 136us) write_op.cc:270] Start()
0623 08:04:39.340409 (+ 34us) write_op.cc:276] Timestamp: P: 1750665870529997 usec, L: 0
0623 08:04:39.340544 (+ 135us) write_op.cc:270] Start()
0623 08:04:39.340563 (+ 19us) write_op.cc:276] Timestamp: P: 1750665870540853 usec, L: 0
0623 08:04:39.340700 (+ 137us) write_op.cc:270] Start()
0623 08:04:39.340721 (+ 21us) write_op.cc:276] Timestamp: P: 1750665870547776 usec, L: 0
0623 08:04:39.340899 (+ 178us) write_op.cc:270] Start()
0623 08:04:39.340918 (+ 19us) write_op.cc:276] Timestamp: P: 1750665870584420 usec, L: 0
0623 08:04:39.341058 (+ 140us) write_op.cc:270] Start()
0623 08:04:39.341077 (+ 19us) write_op.cc:276] Timestamp: P: 1750665870592770 usec, L: 0
0623 08:04:39.341189 (+ 112us) write_op.cc:270] Start()
0623 08:04:39.341208 (+ 19us) write_op.cc:276] Timestamp: P: 1750665870598258 usec, L: 0
0623 08:04:39.341339 (+ 131us) write_op.cc:270] Start()
0623 08:04:39.341359 (+ 20us) write_op.cc:276] Timestamp: P: 1750665870686890 usec, L: 0
0623 08:04:39.341514 (+ 155us) write_op.cc:270] Start()
0623 08:04:39.341534 (+ 20us) write_op.cc:276] Timestamp: P: 1750665870706499 usec, L: 0
0623 08:04:39.341659 (+ 125us) write_op.cc:270] Start()
0623 08:04:39.341680 (+ 21us) write_op.cc:276] Timestamp: P: 1750665870717438 usec, L: 0
0623 08:04:39.341803 (+ 123us) write_op.cc:270] Start()
0623 08:04:39.341822 (+ 19us) write_op.cc:276] Timestamp: P: 1750665870726341 usec, L: 0
0623 08:04:39.341937 (+ 115us) write_op.cc:270] Start()
0623 08:04:39.341957 (+ 20us) write_op.cc:276] Timestamp: P: 1750665870735631 usec, L: 0
0623 08:04:39.342094 (+ 137us) write_op.cc:270] Start()
0623 08:04:39.342120 (+ 26us) write_op.cc:276] Timestamp: P: 1750665870744516 usec, L: 0
0623 08:04:39.342233 (+ 113us) write_op.cc:270] Start()
0623 08:04:39.342248 (+ 15us) write_op.cc:276] Timestamp: P: 1750665870755788 usec, L: 0
0623 08:04:39.342347 (+ 99us) write_op.cc:270] Start()
0623 08:04:39.342364 (+ 17us) write_op.cc:276] Timestamp: P: 1750665870766228 usec, L: 0
0623 08:04:39.342569 (+ 205us) write_op.cc:270] Start()
0623 08:04:39.342590 (+ 21us) write_op.cc:276] Timestamp: P: 1750665870780899 usec, L: 0
0623 08:04:39.342719 (+ 129us) write_op.cc:270] Start()
0623 08:04:39.352317 (+ 9598us) write_op.cc:276] Timestamp: P: 1750665870790229 usec, L: 0
0623 08:04:39.352595 (+ 278us) write_op.cc:270] Start()
0623 08:04:39.352625 (+ 30us) write_op.cc:276] Timestamp: P: 1750665870803711 usec, L: 0
0623 08:04:39.352762 (+ 137us) write_op.cc:270] Start()
0623 08:04:39.352782 (+ 20us) write_op.cc:276] Timestamp: P: 1750665870806975 usec, L: 0
0623 08:04:39.352900 (+ 118us) write_op.cc:270] Start()
0623 08:04:39.352917 (+ 17us) write_op.cc:276] Timestamp: P: 1750665870815892 usec, L: 0
0623 08:04:39.353037 (+ 120us) write_op.cc:270] Start()
0623 08:04:39.353059 (+ 22us) write_op.cc:276] Timestamp: P: 1750665870822154 usec, L: 0
0623 08:04:39.353247 (+ 188us) write_op.cc:270] Start()
0623 08:04:39.353265 (+ 18us) write_op.cc:276] Timestamp: P: 1750665870822945 usec, L: 0
0623 08:04:39.353361 (+ 96us) write_op.cc:270] Start()
0623 08:04:39.353388 (+ 27us) write_op.cc:276] Timestamp: P: 1750665870871776 usec, L: 0
0623 08:04:39.353503 (+ 115us) write_op.cc:270] Start()
0623 08:04:39.353519 (+ 16us) write_op.cc:276] Timestamp: P: 1750665870872847 usec, L: 0
0623 08:04:39.353615 (+ 96us) write_op.cc:270] Start()
0623 08:04:39.353629 (+ 14us) write_op.cc:276] Timestamp: P: 1750665870873495 usec, L: 0
0623 08:04:39.353779 (+ 150us) write_op.cc:270] Start()
0623 08:04:39.353803 (+ 24us) write_op.cc:276] Timestamp: P: 1750665870911596 usec, L: 0
0623 08:04:39.353928 (+ 125us) write_op.cc:270] Start()
0623 08:04:39.353955 (+ 27us) write_op.cc:276] Timestamp: P: 1750665870912589 usec, L: 0
0623 08:04:39.354065 (+ 110us) write_op.cc:270] Start()
0623 08:04:39.354080 (+ 15us) write_op.cc:276] Timestamp: P: 1750665870913266 usec, L: 0
0623 08:04:39.354182 (+ 102us) write_op.cc:270] Start()
0623 08:04:39.354202 (+ 20us) write_op.cc:276] Timestamp: P: 1750665870980302 usec, L: 0
0623 08:04:39.354344 (+ 142us) write_op.cc:270] Start()
0623 08:04:39.354361 (+ 17us) write_op.cc:276] Timestamp: P: 1750665870981100 usec, L: 0
0623 08:04:39.354488 (+ 127us) write_op.cc:270] Start()
0623 08:04:39.354504 (+ 16us) write_op.cc:276] Timestamp: P: 1750665871039112 usec, L: 0
0623 08:04:39.354619 (+ 115us) write_op.cc:270] Start()
0623 08:04:39.354637 (+ 18us) write_op.cc:276] Timestamp: P: 1750665871065435 usec, L: 0
0623 08:04:39.378631 (+ 23994us) write_op.cc:270] Start()
0623 08:04:39.379002 (+ 371us) write_op.cc:276] Timestamp: P: 1750665871089442 usec, L: 0
0623 08:04:39.380746 (+ 1744us) write_op.cc:270] Start()
0623 08:04:39.381509 (+ 763us) write_op.cc:276] Timestamp: P: 1750665871091299 usec, L: 0
0623 08:04:39.387850 (+ 6341us) write_op.cc:270] Start()
0623 08:04:39.388059 (+ 209us) write_op.cc:276] Timestamp: P: 1750665871098643 usec, L: 0
0623 08:04:39.389462 (+ 1403us) write_op.cc:270] Start()
0623 08:04:39.389679 (+ 217us) write_op.cc:276] Timestamp: P: 1750665871115515 usec, L: 0
0623 08:04:39.391290 (+ 1611us) write_op.cc:270] Start()
0623 08:04:39.391505 (+ 215us) write_op.cc:276] Timestamp: P: 1750665871165438 usec, L: 0
0623 08:04:39.392816 (+ 1311us) write_op.cc:270] Start()
0623 08:04:39.392851 (+ 35us) write_op.cc:276] Timestamp: P: 1750665871171686 usec, L: 0
0623 08:04:39.393932 (+ 1081us) write_op.cc:270] Start()
0623 08:04:39.394057 (+ 125us) write_op.cc:276] Timestamp: P: 1750665871188955 usec, L: 0
0623 08:04:39.394988 (+ 931us) write_op.cc:270] Start()
0623 08:04:39.395122 (+ 134us) write_op.cc:276] Timestamp: P: 1750665871193615 usec, L: 0
0623 08:04:39.395840 (+ 718us) write_op.cc:270] Start()
0623 08:04:39.395978 (+ 138us) write_op.cc:276] Timestamp: P: 1750665871237917 usec, L: 0
0623 08:04:39.396647 (+ 669us) write_op.cc:270] Start()
0623 08:04:39.396769 (+ 122us) write_op.cc:276] Timestamp: P: 1750665871244459 usec, L: 0
0623 08:04:39.397428 (+ 659us) write_op.cc:270] Start()
0623 08:04:39.397452 (+ 24us) write_op.cc:276] Timestamp: P: 1750665871248218 usec, L: 0
0623 08:04:39.398094 (+ 642us) write_op.cc:270] Start()
0623 08:04:39.398119 (+ 25us) write_op.cc:276] Timestamp: P: 1750665871334030 usec, L: 0
0623 08:04:39.399007 (+ 888us) write_op.cc:270] Start()
0623 08:04:39.399130 (+ 123us) write_op.cc:276] Timestamp: P: 1750665871361245 usec, L: 0
0623 08:04:39.399868 (+ 738us) write_op.cc:270] Start()
0623 08:04:39.400042 (+ 174us) write_op.cc:276] Timestamp: P: 1750665871370951 usec, L: 0
0623 08:04:39.405795 (+ 5753us) write_op.cc:270] Start()
0623 08:04:39.405927 (+ 132us) write_op.cc:276] Timestamp: P: 1750665871375308 usec, L: 0
0623 08:04:39.406708 (+ 781us) write_op.cc:270] Start()
0623 08:04:39.406875 (+ 167us) write_op.cc:276] Timestamp: P: 1750665871387628 usec, L: 0
0623 08:04:39.408777 (+ 1902us) write_op.cc:270] Start()
0623 08:04:39.408928 (+ 151us) write_op.cc:276] Timestamp: P: 1750665871410200 usec, L: 0
0623 08:04:39.410239 (+ 1311us) write_op.cc:270] Start()
0623 08:04:39.410266 (+ 27us) write_op.cc:276] Timestamp: P: 1750665871412405 usec, L: 0
0623 08:04:39.411449 (+ 1183us) write_op.cc:270] Start()
0623 08:04:39.411480 (+ 31us) write_op.cc:276] Timestamp: P: 1750665871415204 usec, L: 0
0623 08:04:39.413244 (+ 1764us) write_op.cc:270] Start()
0623 08:04:39.413517 (+ 273us) write_op.cc:276] Timestamp: P: 1750665871416639 usec, L: 0
0623 08:04:39.415678 (+ 2161us) write_op.cc:270] Start()
0623 08:04:39.415813 (+ 135us) write_op.cc:276] Timestamp: P: 1750665871422083 usec, L: 0
0623 08:04:39.417776 (+ 1963us) write_op.cc:270] Start()
0623 08:04:39.417903 (+ 127us) write_op.cc:276] Timestamp: P: 1750665871495469 usec, L: 0
0623 08:04:39.419190 (+ 1287us) write_op.cc:270] Start()
0623 08:04:39.419325 (+ 135us) write_op.cc:276] Timestamp: P: 1750665871531769 usec, L: 0
0623 08:04:39.420584 (+ 1259us) write_op.cc:270] Start()
0623 08:04:39.427233 (+ 6649us) write_op.cc:276] Timestamp: P: 1750665871646623 usec, L: 0
0623 08:04:39.428839 (+ 1606us) write_op.cc:270] Start()
0623 08:04:39.429098 (+ 259us) write_op.cc:276] Timestamp: P: 1750665871673537 usec, L: 0
0623 08:04:39.430329 (+ 1231us) write_op.cc:270] Start()
0623 08:04:39.430462 (+ 133us) write_op.cc:276] Timestamp: P: 1750665871702528 usec, L: 0
0623 08:04:39.432044 (+ 1582us) write_op.cc:270] Start()
0623 08:04:39.432507 (+ 463us) write_op.cc:276] Timestamp: P: 1750665871706459 usec, L: 0
0623 08:04:39.433803 (+ 1296us) write_op.cc:270] Start()
0623 08:04:39.433960 (+ 157us) write_op.cc:276] Timestamp: P: 1750665871866987 usec, L: 0
0623 08:04:39.435577 (+ 1617us) write_op.cc:270] Start()
0623 08:04:39.435835 (+ 258us) write_op.cc:276] Timestamp: P: 1750665871867780 usec, L: 0
0623 08:04:39.444453 (+ 8618us) write_op.cc:270] Start()
0623 08:04:39.444489 (+ 36us) write_op.cc:276] Timestamp: P: 1750665871897903 usec, L: 0
0623 08:04:39.444688 (+ 199us) write_op.cc:270] Start()
0623 08:04:39.444716 (+ 28us) write_op.cc:276] Timestamp: P: 1750665871904392 usec, L: 0
0623 08:04:39.445678 (+ 962us) write_op.cc:270] Start()
0623 08:04:39.445706 (+ 28us) write_op.cc:276] Timestamp: P: 1750665871911553 usec, L: 0
0623 08:04:39.445885 (+ 179us) write_op.cc:270] Start()
0623 08:04:39.445910 (+ 25us) write_op.cc:276] Timestamp: P: 1750665871913644 usec, L: 0
0623 08:04:39.446055 (+ 145us) write_op.cc:270] Start()
0623 08:04:39.446081 (+ 26us) write_op.cc:276] Timestamp: P: 1750665871936167 usec, L: 0
0623 08:04:39.446282 (+ 201us) write_op.cc:270] Start()
0623 08:04:39.446306 (+ 24us) write_op.cc:276] Timestamp: P: 1750665871942396 usec, L: 0
0623 08:04:39.446445 (+ 139us) write_op.cc:270] Start()
0623 08:04:39.446476 (+ 31us) write_op.cc:276] Timestamp: P: 1750665871944156 usec, L: 0
0623 08:04:39.446624 (+ 148us) write_op.cc:270] Start()
0623 08:04:39.446646 (+ 22us) write_op.cc:276] Timestamp: P: 1750665871946816 usec, L: 0
0623 08:04:39.446817 (+ 171us) write_op.cc:270] Start()
0623 08:04:39.446843 (+ 26us) write_op.cc:276] Timestamp: P: 1750665871970135 usec, L: 0
0623 08:04:39.446991 (+ 148us) write_op.cc:270] Start()
0623 08:04:39.447012 (+ 21us) write_op.cc:276] Timestamp: P: 1750665871972166 usec, L: 0
0623 08:04:39.447178 (+ 166us) write_op.cc:270] Start()
0623 08:04:39.451232 (+ 4054us) write_op.cc:276] Timestamp: P: 1750665871991404 usec, L: 0
0623 08:04:39.451517 (+ 285us) write_op.cc:270] Start()
0623 08:04:39.451554 (+ 37us) write_op.cc:276] Timestamp: P: 1750665871998941 usec, L: 0
0623 08:04:39.451699 (+ 145us) write_op.cc:270] Start()
0623 08:04:39.451725 (+ 26us) write_op.cc:276] Timestamp: P: 1750665872001010 usec, L: 0
0623 08:04:39.451907 (+ 182us) write_op.cc:270] Start()
0623 08:04:39.451931 (+ 24us) write_op.cc:276] Timestamp: P: 1750665872001582 usec, L: 0
0623 08:04:39.452077 (+ 146us) write_op.cc:270] Start()
0623 08:04:39.452100 (+ 23us) write_op.cc:276] Timestamp: P: 1750665872105329 usec, L: 0
0623 08:04:39.452241 (+ 141us) write_op.cc:270] Start()
0623 08:04:39.452265 (+ 24us) write_op.cc:276] Timestamp: P: 1750665872122667 usec, L: 0
0623 08:04:39.452435 (+ 170us) write_op.cc:270] Start()
0623 08:04:39.452456 (+ 21us) write_op.cc:276] Timestamp: P: 1750665872140559 usec, L: 0
0623 08:04:39.452605 (+ 149us) write_op.cc:270] Start()
0623 08:04:39.452628 (+ 23us) write_op.cc:276] Timestamp: P: 1750665872141264 usec, L: 0
0623 08:04:39.452786 (+ 158us) write_op.cc:270] Start()
0623 08:04:39.452818 (+ 32us) write_op.cc:276] Timestamp: P: 1750665872144610 usec, L: 0
0623 08:04:39.452997 (+ 179us) write_op.cc:270] Start()
0623 08:04:39.453020 (+ 23us) write_op.cc:276] Timestamp: P: 1750665872146524 usec, L: 0
0623 08:04:39.453173 (+ 153us) write_op.cc:270] Start()
0623 08:04:39.453196 (+ 23us) write_op.cc:276] Timestamp: P: 1750665872149131 usec, L: 0
0623 08:04:39.453343 (+ 147us) write_op.cc:270] Start()
0623 08:04:39.453364 (+ 21us) write_op.cc:276] Timestamp: P: 1750665872198526 usec, L: 0
0623 08:04:39.453519 (+ 155us) write_op.cc:270] Start()
0623 08:04:39.453541 (+ 22us) write_op.cc:276] Timestamp: P: 1750665872199321 usec, L: 0
0623 08:04:39.453674 (+ 133us) write_op.cc:270] Start()
0623 08:04:39.453698 (+ 24us) write_op.cc:276] Timestamp: P: 1750665872205772 usec, L: 0
0623 08:04:39.453880 (+ 182us) write_op.cc:270] Start()
0623 08:04:39.453916 (+ 36us) write_op.cc:276] Timestamp: P: 1750665872448278 usec, L: 0
0623 08:04:39.454062 (+ 146us) write_op.cc:270] Start()
0623 08:04:39.454086 (+ 24us) write_op.cc:276] Timestamp: P: 1750665872476980 usec, L: 0
0623 08:04:39.454233 (+ 147us) write_op.cc:270] Start()
0623 08:04:39.454255 (+ 22us) write_op.cc:276] Timestamp: P: 1750665872477778 usec, L: 0
0623 08:04:39.454407 (+ 152us) write_op.cc:270] Start()
0623 08:04:39.454429 (+ 22us) write_op.cc:276] Timestamp: P: 1750665872478345 usec, L: 0
0623 08:04:39.454572 (+ 143us) write_op.cc:270] Start()
0623 08:04:39.454595 (+ 23us) write_op.cc:276] Timestamp: P: 1750665872479006 usec, L: 0
0623 08:04:39.454740 (+ 145us) write_op.cc:270] Start()
0623 08:04:39.454781 (+ 41us) write_op.cc:276] Timestamp: P: 1750665872634408 usec, L: 0
0623 08:04:39.454952 (+ 171us) write_op.cc:270] Start()
0623 08:04:39.454982 (+ 30us) write_op.cc:276] Timestamp: P: 1750665872635505 usec, L: 0
0623 08:04:39.455131 (+ 149us) write_op.cc:270] Start()
0623 08:04:39.455154 (+ 23us) write_op.cc:276] Timestamp: P: 1750665872636339 usec, L: 0
0623 08:04:39.455289 (+ 135us) write_op.cc:270] Start()
0623 08:04:39.455305 (+ 16us) write_op.cc:276] Timestamp: P: 1750665872637313 usec, L: 0
0623 08:04:39.455483 (+ 178us) write_op.cc:270] Start()
0623 08:04:39.455504 (+ 21us) write_op.cc:276] Timestamp: P: 1750665872638233 usec, L: 0
0623 08:04:39.455670 (+ 166us) write_op.cc:270] Start()
0623 08:04:39.455691 (+ 21us) write_op.cc:276] Timestamp: P: 1750665872709163 usec, L: 0
0623 08:04:39.455820 (+ 129us) write_op.cc:270] Start()
0623 08:04:39.455838 (+ 18us) write_op.cc:276] Timestamp: P: 1750665872753885 usec, L: 0
0623 08:04:39.455957 (+ 119us) write_op.cc:270] Start()
0623 08:04:39.455977 (+ 20us) write_op.cc:276] Timestamp: P: 1750665872756506 usec, L: 0
0623 08:04:39.456102 (+ 125us) write_op.cc:270] Start()
0623 08:04:39.456121 (+ 19us) write_op.cc:276] Timestamp: P: 1750665872818032 usec, L: 0
0623 08:04:39.456251 (+ 130us) write_op.cc:270] Start()
0623 08:04:39.456270 (+ 19us) write_op.cc:276] Timestamp: P: 1750665872873249 usec, L: 0
0623 08:04:39.456407 (+ 137us) write_op.cc:270] Start()
0623 08:04:39.456425 (+ 18us) write_op.cc:276] Timestamp: P: 1750665872903651 usec, L: 0
0623 08:04:39.456568 (+ 143us) write_op.cc:270] Start()
0623 08:04:39.456588 (+ 20us) write_op.cc:276] Timestamp: P: 1750665872930886 usec, L: 0
0623 08:04:39.456704 (+ 116us) write_op.cc:270] Start()
0623 08:04:39.456722 (+ 18us) write_op.cc:276] Timestamp: P: 1750665872938676 usec, L: 0
0623 08:04:39.456852 (+ 130us) write_op.cc:270] Start()
0623 08:04:39.456872 (+ 20us) write_op.cc:276] Timestamp: P: 1750665872979208 usec, L: 0
0623 08:04:39.457011 (+ 139us) write_op.cc:270] Start()
0623 08:04:39.457031 (+ 20us) write_op.cc:276] Timestamp: P: 1750665872996388 usec, L: 0
0623 08:04:39.457144 (+ 113us) write_op.cc:270] Start()
0623 08:04:39.457164 (+ 20us) write_op.cc:276] Timestamp: P: 1750665873230677 usec, L: 0
0623 08:04:39.457310 (+ 146us) write_op.cc:270] Start()
0623 08:04:39.457327 (+ 17us) write_op.cc:276] Timestamp: P: 1750665873409628 usec, L: 0
0623 08:04:39.457493 (+ 166us) write_op.cc:270] Start()
0623 08:04:39.457513 (+ 20us) write_op.cc:276] Timestamp: P: 1750665873428970 usec, L: 0
0623 08:04:39.457680 (+ 167us) write_op.cc:270] Start()
0623 08:04:39.457699 (+ 19us) write_op.cc:276] Timestamp: P: 1750665873429776 usec, L: 0
0623 08:04:39.457834 (+ 135us) write_op.cc:270] Start()
0623 08:04:39.457860 (+ 26us) write_op.cc:276] Timestamp: P: 1750665873430520 usec, L: 0
0623 08:04:39.457981 (+ 121us) write_op.cc:270] Start()
0623 08:04:39.458000 (+ 19us) write_op.cc:276] Timestamp: P: 1750665873457453 usec, L: 0
0623 08:04:39.458163 (+ 163us) write_op.cc:270] Start()
0623 08:04:39.458181 (+ 18us) write_op.cc:276] Timestamp: P: 1750665873484898 usec, L: 0
0623 08:04:39.458308 (+ 127us) write_op.cc:270] Start()
0623 08:04:39.458327 (+ 19us) write_op.cc:276] Timestamp: P: 1750665873486006 usec, L: 0
0623 08:04:39.458471 (+ 144us) write_op.cc:270] Start()
0623 08:04:39.458489 (+ 18us) write_op.cc:276] Timestamp: P: 1750665873487007 usec, L: 0
0623 08:04:39.458633 (+ 144us) write_op.cc:270] Start()
0623 08:04:39.458655 (+ 22us) write_op.cc:276] Timestamp: P: 1750665873502811 usec, L: 0
0623 08:04:39.492889 (+ 34234us) write_op.cc:270] Start()
0623 08:04:39.492939 (+ 50us) write_op.cc:276] Timestamp: P: 1750665873525812 usec, L: 0
0623 08:04:39.493167 (+ 228us) write_op.cc:270] Start()
0623 08:04:39.493186 (+ 19us) write_op.cc:276] Timestamp: P: 1750665873528476 usec, L: 0
0623 08:04:39.493301 (+ 115us) write_op.cc:270] Start()
0623 08:04:39.493318 (+ 17us) write_op.cc:276] Timestamp: P: 1750665873702135 usec, L: 0
0623 08:04:39.493451 (+ 133us) write_op.cc:270] Start()
0623 08:04:39.493468 (+ 17us) write_op.cc:276] Timestamp: P: 1750665873703466 usec, L: 0
0623 08:04:39.493587 (+ 119us) write_op.cc:270] Start()
0623 08:04:39.493612 (+ 25us) write_op.cc:276] Timestamp: P: 1750665873704069 usec, L: 0
0623 08:04:39.493772 (+ 160us) write_op.cc:270] Start()
0623 08:04:39.493796 (+ 24us) write_op.cc:276] Timestamp: P: 1750665873704586 usec, L: 0
0623 08:04:39.493963 (+ 167us) write_op.cc:270] Start()
0623 08:04:39.493987 (+ 24us) write_op.cc:276] Timestamp: P: 1750665873744004 usec, L: 0
0623 08:04:39.494152 (+ 165us) write_op.cc:270] Start()
0623 08:04:39.494179 (+ 27us) write_op.cc:276] Timestamp: P: 1750665873748170 usec, L: 0
0623 08:04:39.499683 (+ 5504us) write_op.cc:270] Start()
0623 08:04:39.499716 (+ 33us) write_op.cc:276] Timestamp: P: 1750665873751251 usec, L: 0
0623 08:04:39.499910 (+ 194us) write_op.cc:270] Start()
0623 08:04:39.499937 (+ 27us) write_op.cc:276] Timestamp: P: 1750665873776968 usec, L: 0
0623 08:04:39.500115 (+ 178us) write_op.cc:270] Start()
0623 08:04:39.500140 (+ 25us) write_op.cc:276] Timestamp: P: 1750665873780420 usec, L: 0
0623 08:04:39.500298 (+ 158us) write_op.cc:270] Start()
0623 08:04:39.500315 (+ 17us) write_op.cc:276] Timestamp: P: 1750665873784542 usec, L: 0
0623 08:04:39.500500 (+ 185us) write_op.cc:270] Start()
0623 08:04:39.500524 (+ 24us) write_op.cc:276] Timestamp: P: 1750665873936139 usec, L: 0
0623 08:04:39.500674 (+ 150us) write_op.cc:270] Start()
0623 08:04:39.500698 (+ 24us) write_op.cc:276] Timestamp: P: 1750665873936808 usec, L: 0
0623 08:04:39.500839 (+ 141us) write_op.cc:270] Start()
0623 08:04:39.500862 (+ 23us) write_op.cc:276] Timestamp: P: 1750665874365607 usec, L: 0
0623 08:04:39.501018 (+ 156us) write_op.cc:270] Start()
0623 08:04:39.501041 (+ 23us) write_op.cc:276] Timestamp: P: 1750665874375012 usec, L: 0
0623 08:04:39.501215 (+ 174us) write_op.cc:270] Start()
0623 08:04:39.501240 (+ 25us) write_op.cc:276] Timestamp: P: 1750665874381074 usec, L: 0
0623 08:04:39.501377 (+ 137us) write_op.cc:270] Start()
0623 08:04:39.501413 (+ 36us) write_op.cc:276] Timestamp: P: 1750665874401627 usec, L: 0
0623 08:04:39.501561 (+ 148us) write_op.cc:270] Start()
0623 08:04:39.501584 (+ 23us) write_op.cc:276] Timestamp: P: 1750665874422587 usec, L: 0
0623 08:04:39.501722 (+ 138us) write_op.cc:270] Start()
0623 08:04:39.501744 (+ 22us) write_op.cc:276] Timestamp: P: 1750665874423517 usec, L: 0
0623 08:04:39.510709 (+ 8965us) write_op.cc:270] Start()
0623 08:04:39.510744 (+ 35us) write_op.cc:276] Timestamp: P: 1750665874458620 usec, L: 0
0623 08:04:39.510942 (+ 198us) write_op.cc:270] Start()
0623 08:04:39.510962 (+ 20us) write_op.cc:276] Timestamp: P: 1750665874460585 usec, L: 0
0623 08:04:39.511081 (+ 119us) write_op.cc:270] Start()
0623 08:04:39.511099 (+ 18us) write_op.cc:276] Timestamp: P: 1750665874462102 usec, L: 0
0623 08:04:39.511225 (+ 126us) write_op.cc:270] Start()
0623 08:04:39.511243 (+ 18us) write_op.cc:276] Timestamp: P: 1750665874514425 usec, L: 0
0623 08:04:39.511361 (+ 118us) write_op.cc:270] Start()
0623 08:04:39.511382 (+ 21us) write_op.cc:276] Timestamp: P: 1750665874546347 usec, L: 0
0623 08:04:39.511501 (+ 119us) write_op.cc:270] Start()
0623 08:04:39.511519 (+ 18us) write_op.cc:276] Timestamp: P: 1750665874549352 usec, L: 0
0623 08:04:39.592656 (+ 81137us) log.cc:844] Serialized 116122 byte log entry
0623 08:04:39.597310 (+ 4654us) raft_consensus.cc:1643] Marking committed up to 516
0623 08:04:39.597499 (+ 189us) log.cc:844] Serialized 15 byte log entry
0623 08:04:39.617290 (+ 19791us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.617296 (+ 6us) write_op.cc:454] Released schema lock
0623 08:04:39.625673 (+ 8377us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.625681 (+ 8us) write_op.cc:454] Released schema lock
0623 08:04:39.626400 (+ 719us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.626404 (+ 4us) write_op.cc:454] Released schema lock
0623 08:04:39.627179 (+ 775us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.627184 (+ 5us) write_op.cc:454] Released schema lock
0623 08:04:39.637158 (+ 9974us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.637165 (+ 7us) write_op.cc:454] Released schema lock
0623 08:04:39.648243 (+ 11078us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.648249 (+ 6us) write_op.cc:454] Released schema lock
0623 08:04:39.649031 (+ 782us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.649036 (+ 5us) write_op.cc:454] Released schema lock
0623 08:04:39.649666 (+ 630us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.649671 (+ 5us) write_op.cc:454] Released schema lock
0623 08:04:39.650227 (+ 556us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.650232 (+ 5us) write_op.cc:454] Released schema lock
0623 08:04:39.651343 (+ 1111us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.651348 (+ 5us) write_op.cc:454] Released schema lock
0623 08:04:39.657899 (+ 6551us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.657907 (+ 8us) write_op.cc:454] Released schema lock
0623 08:04:39.659072 (+ 1165us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.659076 (+ 4us) write_op.cc:454] Released schema lock
0623 08:04:39.660102 (+ 1026us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.660109 (+ 7us) write_op.cc:454] Released schema lock
0623 08:04:39.660922 (+ 813us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.660927 (+ 5us) write_op.cc:454] Released schema lock
0623 08:04:39.661590 (+ 663us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.661598 (+ 8us) write_op.cc:454] Released schema lock
0623 08:04:39.662328 (+ 730us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.662335 (+ 7us) write_op.cc:454] Released schema lock
0623 08:04:39.683749 (+ 21414us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.683757 (+ 8us) write_op.cc:454] Released schema lock
0623 08:04:39.684912 (+ 1155us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.684920 (+ 8us) write_op.cc:454] Released schema lock
0623 08:04:39.686937 (+ 2017us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.686946 (+ 9us) write_op.cc:454] Released schema lock
0623 08:04:39.687815 (+ 869us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.687822 (+ 7us) write_op.cc:454] Released schema lock
0623 08:04:39.688469 (+ 647us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.688472 (+ 3us) write_op.cc:454] Released schema lock
0623 08:04:39.689088 (+ 616us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.689091 (+ 3us) write_op.cc:454] Released schema lock
0623 08:04:39.692818 (+ 3727us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.692823 (+ 5us) write_op.cc:454] Released schema lock
0623 08:04:39.693905 (+ 1082us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.693910 (+ 5us) write_op.cc:454] Released schema lock
0623 08:04:39.694540 (+ 630us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.694545 (+ 5us) write_op.cc:454] Released schema lock
0623 08:04:39.695342 (+ 797us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.695349 (+ 7us) write_op.cc:454] Released schema lock
0623 08:04:39.696902 (+ 1553us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.696910 (+ 8us) write_op.cc:454] Released schema lock
0623 08:04:39.698467 (+ 1557us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.698474 (+ 7us) write_op.cc:454] Released schema lock
0623 08:04:39.699989 (+ 1515us) write_op.cc:489] Releasing partition, row and schema l
I20250623 08:04:40.454993 1507 raft_consensus.cc:513] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.455603 1507 leader_election.cc:290] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.459007 32314 leader_election.cc:304] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:40.459230 32314 raft_consensus.cc:2684] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250623 08:04:40.459510 32314 leader_election.cc:343] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 18ea63263b9b466e922169eecafab747
W20250623 08:04:40.461751 32453 leader_election.cc:343] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.462284 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 52d0a356c74f4daea9093c998b4bf73e (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
W20250623 08:04:40.464370 32453 leader_election.cc:343] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.464495 32453 leader_election.cc:304] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.464705 1659 raft_consensus.cc:2747] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:40.464838 32453 leader_election.cc:343] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:40.465131 32453 leader_election.cc:343] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.467238 339 tablet_bootstrap.cc:492] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:40.467427 339 ts_tablet_manager.cc:1397] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.001s
I20250623 08:04:40.468173 339 raft_consensus.cc:357] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.468426 339 raft_consensus.cc:383] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.468525 339 raft_consensus.cc:738] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.468766 339 consensus_queue.cc:260] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.469834 339 ts_tablet_manager.cc:1428] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:40.470487 339 tablet_bootstrap.cc:492] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:40.472191 339 tablet_bootstrap.cc:654] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.473301 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 035bb06bc7dd42f681886def9d56eb73 (DEFAULT_TABLE table=default.Table_905d0330c27149a19dabb586c51a2e0a [id=b3f407a7a26e4bd5ae9a719a5c4577c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.473733 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 035bb06bc7dd42f681886def9d56eb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.480257 1806 consensus_queue.cc:1035] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.480733 1659 consensus_queue.cc:1035] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.481007 342 raft_consensus.cc:357] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.481177 342 raft_consensus.cc:383] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.481238 342 raft_consensus.cc:738] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.481477 342 consensus_queue.cc:260] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.482458 342 ts_tablet_manager.cc:1428] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.060s user 0.001s sys 0.000s
I20250623 08:04:40.482833 342 tablet_bootstrap.cc:492] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:40.484792 342 tablet_bootstrap.cc:654] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.486315 952 ts_tablet_manager.cc:1918] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:40.486459 952 log.cc:1199] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/36c8714cfbd84dd38cf5c0f305ef6843
I20250623 08:04:40.486827 952 ts_tablet_manager.cc:1939] T 36c8714cfbd84dd38cf5c0f305ef6843 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:40.460666 343 raft_consensus.cc:357] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.488756 343 raft_consensus.cc:383] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.488916 343 raft_consensus.cc:738] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.489409 343 consensus_queue.cc:260] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.490139 343 ts_tablet_manager.cc:1428] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20250623 08:04:40.472916 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 035bb06bc7dd42f681886def9d56eb73 (DEFAULT_TABLE table=default.Table_905d0330c27149a19dabb586c51a2e0a [id=b3f407a7a26e4bd5ae9a719a5c4577c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.490669 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 035bb06bc7dd42f681886def9d56eb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.479390 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 (DEFAULT_TABLE table=default.Table_2ecc70ddf2dc40e081fefb676d46a68d [id=3a1a4d6d2cc24dc8abe85a4597c0d0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.495225 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef27bb5b1764ef09888fb0dfcae5be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.462841 32589 leader_election.cc:304] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.499419 32589 leader_election.cc:304] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.498975 1789 raft_consensus.cc:491] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.499871 1789 raft_consensus.cc:513] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.500505 1789 leader_election.cc:290] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.500771 1707 raft_consensus.cc:2802] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:40.500870 1707 raft_consensus.cc:695] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:40.501101 1707 consensus_queue.cc:237] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.502035 1771 raft_consensus.cc:2802] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:40.504307 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.508077 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 1ed334ed653d4dea827f4e6ce9e1bd8b (DEFAULT_TABLE table=default.Table_f0683327c2bf468582a2f3e557d15725 [id=acb1a73cc8b546dab7431a57c3a8857e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.508498 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ed334ed653d4dea827f4e6ce9e1bd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.508466 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 1ed334ed653d4dea827f4e6ce9e1bd8b (DEFAULT_TABLE table=default.Table_f0683327c2bf468582a2f3e557d15725 [id=acb1a73cc8b546dab7431a57c3a8857e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.508872 32627 tablet_service.cc:1468] Processing CreateTablet for tablet d5f82dc198af45f598bf05c05a4a8090 (DEFAULT_TABLE table=default.Table_1a1b8de8d0a645e9a48938956c562be9 [id=2562e6968e504a5aa52358337d79f528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.508980 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ed334ed653d4dea827f4e6ce9e1bd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.509208 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f82dc198af45f598bf05c05a4a8090. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.510859 32353 tablet_service.cc:1468] Processing CreateTablet for tablet d5f82dc198af45f598bf05c05a4a8090 (DEFAULT_TABLE table=default.Table_1a1b8de8d0a645e9a48938956c562be9 [id=2562e6968e504a5aa52358337d79f528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.511202 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f82dc198af45f598bf05c05a4a8090. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.511982 32634 tablet_service.cc:1468] Processing CreateTablet for tablet 1793340e45b74e4aba3f57c20cc54903 (DEFAULT_TABLE table=default.Table_8ab4ba6029964c2fa123202d6c5c836d [id=3c1572742d694d48be03ccabe9166f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.512135 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 4a0615e6f8c346aba5e5f30362bc6de4 (DEFAULT_TABLE table=default.Table_738cccce8e6c47d683b9a7c569aa58e3 [id=20baa502b37c4ec1904f52ddeaf99c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.512303 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1793340e45b74e4aba3f57c20cc54903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.513059 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a0615e6f8c346aba5e5f30362bc6de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.513123 32352 tablet_service.cc:1468] Processing CreateTablet for tablet 79014324599840038cf53ca02cbc6f7e (DEFAULT_TABLE table=default.Table_ff6e3cec929a433b8ac68152b9095dfc [id=ce7434eee7664a8686eb22f28df7f1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.513427 32352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79014324599840038cf53ca02cbc6f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.515002 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 4a0615e6f8c346aba5e5f30362bc6de4 (DEFAULT_TABLE table=default.Table_738cccce8e6c47d683b9a7c569aa58e3 [id=20baa502b37c4ec1904f52ddeaf99c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.515002 32622 tablet_service.cc:1468] Processing CreateTablet for tablet 79014324599840038cf53ca02cbc6f7e (DEFAULT_TABLE table=default.Table_ff6e3cec929a433b8ac68152b9095dfc [id=ce7434eee7664a8686eb22f28df7f1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.515014 32351 tablet_service.cc:1468] Processing CreateTablet for tablet 04fde7d1c1744c0597e43086ca4c9541 (DEFAULT_TABLE table=default.Table_06b731510ae743608f1fbc4d250acf14 [id=7fbe15151173464ea0ee480720c58b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.515312 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79014324599840038cf53ca02cbc6f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.515336 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a0615e6f8c346aba5e5f30362bc6de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.515576 32351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fde7d1c1744c0597e43086ca4c9541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.515806 32349 tablet_service.cc:1468] Processing CreateTablet for tablet 1793340e45b74e4aba3f57c20cc54903 (DEFAULT_TABLE table=default.Table_8ab4ba6029964c2fa123202d6c5c836d [id=3c1572742d694d48be03ccabe9166f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.516103 32349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1793340e45b74e4aba3f57c20cc54903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.519001 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 04fde7d1c1744c0597e43086ca4c9541 (DEFAULT_TABLE table=default.Table_06b731510ae743608f1fbc4d250acf14 [id=7fbe15151173464ea0ee480720c58b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.519345 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fde7d1c1744c0597e43086ca4c9541. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:40.529932 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 49231.1us
I20250623 08:04:40.530238 1771 raft_consensus.cc:695] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
W20250623 08:04:40.530341 32453 leader_election.cc:343] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.530483 1507 raft_consensus.cc:491] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.530582 1507 raft_consensus.cc:513] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.530530 1771 consensus_queue.cc:237] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.531105 1843 raft_consensus.cc:491] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.531219 1507 leader_election.cc:290] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.531222 1843 raft_consensus.cc:513] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.531747 1843 leader_election.cc:290] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.531901 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6926c49b787941d98c647543bd3b9ed5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.532061 32379 raft_consensus.cc:2466] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:40.532565 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6926c49b787941d98c647543bd3b9ed5" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.532756 32649 raft_consensus.cc:2466] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:40.533115 32314 leader_election.cc:304] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:40.533218 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.533577 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:40.533596 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6926c49b787941d98c647543bd3b9ed5" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.533780 32655 raft_consensus.cc:2391] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:40.533746 32516 raft_consensus.cc:2466] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:40.533916 1843 raft_consensus.cc:2802] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:40.533995 1843 raft_consensus.cc:491] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.534060 1843 raft_consensus.cc:3058] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.535100 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8de2b1105564fdab386442b1de4cedb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.535817 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6926c49b787941d98c647543bd3b9ed5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.536145 32646 raft_consensus.cc:2466] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:40.536563 1805 tablet.cc:1717] T cdc9d47f2232403fa71e14a6fe6f33c3 P 125745849172449ba4be8244d61fdcb3: Skipping requested alter to schema version 2, tablet already version 2
I20250623 08:04:40.537361 1843 raft_consensus.cc:513] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:40.539058 32589 leader_election.cc:343] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.539381 1843 leader_election.cc:290] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.539945 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6926c49b787941d98c647543bd3b9ed5" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:40.540113 32516 raft_consensus.cc:3058] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.540375 32589 leader_election.cc:304] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:40.541308 32589 leader_election.cc:343] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.539378 1707 raft_consensus.cc:491] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.542570 1707 raft_consensus.cc:513] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.543253 1707 leader_election.cc:290] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.543381 1707 raft_consensus.cc:2747] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.543394 32516 raft_consensus.cc:2466] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:40.543678 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6926c49b787941d98c647543bd3b9ed5" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:40.543831 32647 raft_consensus.cc:3058] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.543954 32314 leader_election.cc:304] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:40.544723 1843 raft_consensus.cc:2802] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:40.545049 1843 raft_consensus.cc:695] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:40.545301 1843 consensus_queue.cc:237] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.547859 32647 raft_consensus.cc:2466] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:40.502717 1706 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d31835ec4ad74332a00998d431492466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:40.549589 1706 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466 [sys.catalog]: This master's current role is: LEADER
I20250623 08:04:40.538772 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:40.566455 32589 leader_election.cc:304] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.571254 32589 leader_election.cc:304] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.571702 1771 raft_consensus.cc:2747] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.572453 343 tablet_bootstrap.cc:492] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:40.575791 32589 leader_election.cc:343] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.576473 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:40.576545 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 177) took 5383 ms. Trace:
I20250623 08:04:40.576628 952 rpcz_store.cc:276] 0623 08:04:35.192912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.192975 (+ 63us) service_pool.cc:225] Handling call
0623 08:04:40.317957 (+5124982us) ts_tablet_manager.cc:1145] Deleting tablet 36c8714cfbd84dd38cf5c0f305ef6843
0623 08:04:40.317968 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.437842 (+119874us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.486186 (+ 48344us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.576524 (+ 90338us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52571,"tablet-delete.queue_time_us":5100208}
I20250623 08:04:40.577037 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 178) took 5290 ms. Trace:
I20250623 08:04:40.577139 952 rpcz_store.cc:276] 0623 08:04:35.291534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.291585 (+ 51us) service_pool.cc:225] Handling call
0623 08:04:40.576907 (+5285322us) ts_tablet_manager.cc:1145] Deleting tablet c6a97b34e386443fa19a9601621b5017
0623 08:04:40.576917 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.576997 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5285011}
I20250623 08:04:40.577224 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:40.577492 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 179) took 5283 ms. Trace:
I20250623 08:04:40.577598 952 rpcz_store.cc:276] 0623 08:04:35.293700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.296418 (+ 2718us) service_pool.cc:225] Handling call
0623 08:04:40.577404 (+5280986us) ts_tablet_manager.cc:1145] Deleting tablet ea702abb31fb41498ab61bbb05b1f7e2
0623 08:04:40.577412 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.577476 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5280237}
I20250623 08:04:40.577886 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 180) took 5281 ms. Trace:
I20250623 08:04:40.578250 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]). Expected version 2 got 0
I20250623 08:04:40.577966 952 rpcz_store.cc:276] 0623 08:04:35.296738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.296792 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:40.577823 (+5281031us) ts_tablet_manager.cc:1145] Deleting tablet 2ecf4af9e476495683bd63cac20d4672
0623 08:04:40.577831 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.577876 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5280608}
I20250623 08:04:40.578570 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:40.578609 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 183) took 5278 ms. Trace:
I20250623 08:04:40.578675 952 rpcz_store.cc:276] 0623 08:04:35.299730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.300451 (+ 721us) service_pool.cc:225] Handling call
0623 08:04:40.578532 (+5278081us) ts_tablet_manager.cc:1145] Deleting tablet 4cb66c7a273a412db3ae77af9e30d036
0623 08:04:40.578540 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.578595 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5277886}
I20250623 08:04:40.578984 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 184) took 5278 ms. Trace:
I20250623 08:04:40.579056 952 rpcz_store.cc:276] 0623 08:04:35.300818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.300858 (+ 40us) service_pool.cc:225] Handling call
0623 08:04:40.578907 (+5278049us) ts_tablet_manager.cc:1145] Deleting tablet 279624b8bb834b95a4f08045968ebfb4
0623 08:04:40.578914 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.578968 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5277832}
I20250623 08:04:40.579280 952 tablet_replica.cc:331] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:40.579253 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:40.579464 952 raft_consensus.cc:2241] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.579483 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:40.579636 952 raft_consensus.cc:2270] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.580052 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:40.580176 952 ts_tablet_manager.cc:1905] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:40.582453 32589 leader_election.cc:343] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.582811 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:40.583103 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:40.583729 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:40.583889 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet d5cd316e6f4447be958611ab07ed31fc (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:40.583803 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d1ec9d56ba40128325a50f204fe920" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.584010 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:40.576063 1771 raft_consensus.cc:2802] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:40.584331 1771 raft_consensus.cc:491] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.584446 1771 raft_consensus.cc:3058] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.585023 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]). Expected version 1 got 0
I20250623 08:04:40.585268 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_217651efa5dc4571b2a47cd3196c9e57 [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 1 got 0
I20250623 08:04:40.585539 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:40.585868 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:40.586076 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:40.586222 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:40.586344 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:40.589864 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 (DEFAULT_TABLE table=default.Table_2ecc70ddf2dc40e081fefb676d46a68d [id=3a1a4d6d2cc24dc8abe85a4597c0d0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.590142 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
W20250623 08:04:40.590636 32589 leader_election.cc:343] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.590905 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:40.591136 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 1c529e826fa84a919c018887620e4023 (table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]). Expected version 1 got 0
I20250623 08:04:40.591451 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:40.593461 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:40.594694 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:40.594920 32516 raft_consensus.cc:2391] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:40.595002 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef27bb5b1764ef09888fb0dfcae5be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.595620 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8de2b1105564fdab386442b1de4cedb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.595793 32516 raft_consensus.cc:2466] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:40.596588 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:40.596853 343 tablet_bootstrap.cc:654] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.596870 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:40.597090 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:40.597306 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a9ea74d20c5f4eefbcb8508ed1774026 (table default.Table_0b9c0d17ddcd422eb8035e69884f3da6 [id=ced0b4cf62924dbfadad02109349bea0]). Expected version 1 got 0
I20250623 08:04:40.597748 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:40.598053 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 7419cb91360f49e1b3cb7e90732558c5 (table default.Table_161ad5ddd1cb40e0989e5c237a205db1 [id=7cc0919082d04fc4a02aa856a42e5529]). Expected version 1 got 0
I20250623 08:04:40.598541 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:40.599185 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]). Expected version 1 got 0
I20250623 08:04:40.602865 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d1ec9d56ba40128325a50f204fe920" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.603102 32051 catalog_manager.cc:5582] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:40.603531 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:40.603957 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:40.604223 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:40.605278 342 tablet_bootstrap.cc:492] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:40.605742 32051 catalog_manager.cc:5582] T 21080ab069d74457a9f7fe3aabdb3c7c P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:40.596539 1771 raft_consensus.cc:513] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.606848 1771 leader_election.cc:290] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.607121 342 ts_tablet_manager.cc:1397] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.000s
I20250623 08:04:40.607554 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "789d1c20aa72480ab71123c799bbde4c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.607928 342 raft_consensus.cc:357] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.608086 342 raft_consensus.cc:383] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.608144 342 raft_consensus.cc:738] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.608664 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 5 got 0
I20250623 08:04:40.609228 32051 catalog_manager.cc:5582] T 5f1fb8231e8e45208ac4f412134d8696 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:40.609598 342 consensus_queue.cc:260] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.610121 342 ts_tablet_manager.cc:1428] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:40.610329 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:40.613148 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:40.613484 342 tablet_bootstrap.cc:492] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:40.613787 32051 catalog_manager.cc:5582] T d98654c29b9c4e75af6a213cafd4c2c9 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:40.615365 32051 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:40.615609 1843 raft_consensus.cc:491] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.615780 1843 raft_consensus.cc:513] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:40.616081 32057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4eebc32fdfa44abcbd6f04a636d02c7c consensus state has no leader
I20250623 08:04:40.616256 32057 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 375) took 1208 ms. Trace:
I20250623 08:04:40.616410 1843 leader_election.cc:290] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.616357 32057 rpcz_store.cc:276] 0623 08:04:39.407887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.732746 (+324859us) service_pool.cc:225] Handling call
0623 08:04:39.769842 (+ 37096us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fb5a6 00007f19c7646616 00007f19c7646345 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:39.770054 (+ 212us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:39.770232 (+ 178us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:39.770400 (+ 168us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:39.770472 (+ 72us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:39.860868 (+ 90396us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:40.616222 (+755354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:39.789570 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:39.789663 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:39.789668 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:39.789671 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:39.789752 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:39.789760 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:39.789765 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:39.789810 (+ 45us) tablet.cc:699] Row locks acquired
0623 08:04:39.789812 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:39.789840 (+ 28us) write_op.cc:270] Start()
0623 08:04:39.789864 (+ 24us) write_op.cc:276] Timestamp: P: 1750665879789778 usec, L: 0
0623 08:04:39.789868 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:39.790009 (+ 141us) log.cc:844] Serialized 405 byte log entry
0623 08:04:39.855489 (+ 65480us) op_driver.cc:464] REPLICATION: finished
0623 08:04:39.857642 (+ 2153us) write_op.cc:301] APPLY: starting
0623 08:04:39.857650 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:39.857673 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:39.857691 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:39.857716 (+ 25us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:39.857736 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:39.857742 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:39.857779 (+ 37us) log.cc:844] Serialized 23 byte log entry
0623 08:04:39.857807 (+ 28us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.857818 (+ 11us) write_op.cc:454] Released schema lock
0623 08:04:39.857822 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:39.861648 (+ 3826us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.861665 (+ 17us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":84825984,"child_traces":[["op",{"apply.queue_time_us":2126,"apply.run_cpu_time_us":220,"apply.run_wall_time_us":219,"num_ops":1,"prepare.queue_time_us":18731,"prepare.run_cpu_time_us":597,"prepare.run_wall_time_us":72070,"raft.queue_time_us":43,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"replication_time_us":65607}]]}
I20250623 08:04:40.617901 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7874a1d5cb9a464294ca32b2767efbf3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.618089 32516 raft_consensus.cc:2391] T 7874a1d5cb9a464294ca32b2767efbf3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:40.622630 342 tablet_bootstrap.cc:654] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:40.622978 32053 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c337a37fcb141bca87212317d750de8 consensus state has no leader
I20250623 08:04:40.623139 32053 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 374) took 1222 ms. Trace:
I20250623 08:04:40.623231 32053 rpcz_store.cc:276] 0623 08:04:39.400484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.731352 (+330868us) service_pool.cc:225] Handling call
0623 08:04:39.767278 (+ 35926us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fb5a6 00007f19c7646616 00007f19c7646345 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:39.767481 (+ 203us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:39.767654 (+ 173us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:39.767801 (+ 147us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:39.767862 (+ 61us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:39.860780 (+ 92918us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:40.623108 (+762328us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:39.786556 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:39.786664 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:39.786669 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:39.786671 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:39.786793 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:39.786805 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:39.786812 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:39.788533 (+ 1721us) tablet.cc:699] Row locks acquired
0623 08:04:39.788536 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:39.788572 (+ 36us) write_op.cc:270] Start()
0623 08:04:39.788601 (+ 29us) write_op.cc:276] Timestamp: P: 1750665879788508 usec, L: 0
0623 08:04:39.788605 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:39.788768 (+ 163us) log.cc:844] Serialized 405 byte log entry
0623 08:04:39.855455 (+ 66687us) op_driver.cc:464] REPLICATION: finished
0623 08:04:39.857334 (+ 1879us) write_op.cc:301] APPLY: starting
0623 08:04:39.857342 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:39.857363 (+ 21us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:39.857400 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:39.857423 (+ 23us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:39.857443 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:39.857450 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:39.857485 (+ 35us) log.cc:844] Serialized 23 byte log entry
0623 08:04:39.857532 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.857541 (+ 9us) write_op.cc:454] Released schema lock
0623 08:04:39.857546 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:39.857592 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.857596 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":81959296,"child_traces":[["op",{"apply.queue_time_us":1854,"apply.run_cpu_time_us":252,"apply.run_wall_time_us":251,"num_ops":1,"prepare.queue_time_us":18316,"prepare.run_cpu_time_us":650,"prepare.run_wall_time_us":2973,"replication_time_us":66835}]]}
W20250623 08:04:40.608458 32589 leader_election.cc:343] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.617331 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7874a1d5cb9a464294ca32b2767efbf3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.627420 32314 leader_election.cc:304] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:40.627816 1843 raft_consensus.cc:3058] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.630887 1843 raft_consensus.cc:2747] T 7874a1d5cb9a464294ca32b2767efbf3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:40.634464 32045 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 16bc969d09f74b0986070a504f9f670e consensus state has no leader
W20250623 08:04:40.634967 32054 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee82db25aa544a21a1419e081196f179 consensus state has no leader
I20250623 08:04:40.636454 32050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet a383d10d53b04d5699dd3c4bad8878c7
I20250623 08:04:40.637269 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_5cfbfaf89e6a4bdf94cef7019cf9c5e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:40.637512 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet a383d10d53b04d5699dd3c4bad8878c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
W20250623 08:04:40.637861 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cfbfaf89e6a4bdf94cef7019cf9c5e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:40.638338 981 ts_tablet_manager.cc:1918] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:40.638450 981 log.cc:1199] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/18ea63263b9b466e922169eecafab747
I20250623 08:04:40.638840 981 ts_tablet_manager.cc:1939] T 18ea63263b9b466e922169eecafab747 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:40.640923 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet a383d10d53b04d5699dd3c4bad8878c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:40.641010 32484 tablet_service.cc:1468] Processing CreateTablet for tablet 035bb06bc7dd42f681886def9d56eb73 (DEFAULT_TABLE table=default.Table_905d0330c27149a19dabb586c51a2e0a [id=b3f407a7a26e4bd5ae9a719a5c4577c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.641398 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 035bb06bc7dd42f681886def9d56eb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.641345 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 (DEFAULT_TABLE table=default.Table_2ecc70ddf2dc40e081fefb676d46a68d [id=3a1a4d6d2cc24dc8abe85a4597c0d0c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.641651 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef27bb5b1764ef09888fb0dfcae5be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.648538 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 1ed334ed653d4dea827f4e6ce9e1bd8b (DEFAULT_TABLE table=default.Table_f0683327c2bf468582a2f3e557d15725 [id=acb1a73cc8b546dab7431a57c3a8857e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.648945 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ed334ed653d4dea827f4e6ce9e1bd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.650197 1771 raft_consensus.cc:491] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.650347 1771 raft_consensus.cc:513] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.650998 1771 leader_election.cc:290] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.651471 32485 tablet_service.cc:1468] Processing CreateTablet for tablet d5f82dc198af45f598bf05c05a4a8090 (DEFAULT_TABLE table=default.Table_1a1b8de8d0a645e9a48938956c562be9 [id=2562e6968e504a5aa52358337d79f528]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.651844 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5f82dc198af45f598bf05c05a4a8090. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.658519 32492 tablet_service.cc:1468] Processing CreateTablet for tablet 79014324599840038cf53ca02cbc6f7e (DEFAULT_TABLE table=default.Table_ff6e3cec929a433b8ac68152b9095dfc [id=ce7434eee7664a8686eb22f28df7f1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.658895 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79014324599840038cf53ca02cbc6f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.666309 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_6b3d1637c4c1443089f04128e190256f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:40.666934 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b3d1637c4c1443089f04128e190256f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:40.667797 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 221) took 4671 ms. Trace:
I20250623 08:04:40.667908 981 rpcz_store.cc:276] 0623 08:04:35.996279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.996508 (+ 229us) service_pool.cc:225] Handling call
0623 08:04:40.409224 (+4412716us) ts_tablet_manager.cc:1145] Deleting tablet 18ea63263b9b466e922169eecafab747
0623 08:04:40.409231 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.425068 (+ 15837us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.638200 (+213132us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.667767 (+ 29567us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11476,"tablet-delete.queue_time_us":4406246}
I20250623 08:04:40.668507 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 18ea63263b9b466e922169eecafab747 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]) successfully deleted
I20250623 08:04:40.668820 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 4a0615e6f8c346aba5e5f30362bc6de4 (DEFAULT_TABLE table=default.Table_738cccce8e6c47d683b9a7c569aa58e3 [id=20baa502b37c4ec1904f52ddeaf99c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.669167 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a0615e6f8c346aba5e5f30362bc6de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.676671 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 222) took 4636 ms. Trace:
I20250623 08:04:40.676820 981 rpcz_store.cc:276] 0623 08:04:36.040093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.040194 (+ 101us) service_pool.cc:225] Handling call
0623 08:04:40.676450 (+4636256us) ts_tablet_manager.cc:1145] Deleting tablet 89bb7ea31b4d4ce7a3c063853142fbb2
0623 08:04:40.676466 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.676624 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4635956}
W20250623 08:04:40.677599 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89bb7ea31b4d4ce7a3c063853142fbb2 already in progress: creating tablet
I20250623 08:04:40.679304 32484 tablet_service.cc:1468] Processing CreateTablet for tablet 1793340e45b74e4aba3f57c20cc54903 (DEFAULT_TABLE table=default.Table_8ab4ba6029964c2fa123202d6c5c836d [id=3c1572742d694d48be03ccabe9166f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.679703 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1793340e45b74e4aba3f57c20cc54903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.681679 343 tablet_bootstrap.cc:492] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:40.681913 343 ts_tablet_manager.cc:1397] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.001s
I20250623 08:04:40.647742 1843 raft_consensus.cc:491] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.683451 1843 raft_consensus.cc:513] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.655644 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9049e3d9b39f420bb51221808409fc54" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.684515 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71fadb10e6134478b013358bba056389" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:40.684689 32589 leader_election.cc:343] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.683046 343 raft_consensus.cc:357] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:40.685048 343 raft_consensus.cc:383] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.685139 343 raft_consensus.cc:738] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.685431 343 consensus_queue.cc:260] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:40.685998 343 ts_tablet_manager.cc:1428] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:40.686393 981 tablet_replica.cc:331] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:40.686815 981 raft_consensus.cc:2241] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.687094 981 raft_consensus.cc:2270] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.687872 981 ts_tablet_manager.cc:1905] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.700176 979 ts_tablet_manager.cc:1918] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:40.700887 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71fadb10e6134478b013358bba056389" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.701717 32314 leader_election.cc:304] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:40.702081 1682 raft_consensus.cc:2747] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.702600 1843 leader_election.cc:290] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.710312 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet a383d10d53b04d5699dd3c4bad8878c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:40.711038 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73453eb1a0394cfe8d49adcc71d4f3e9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.712954 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 36c8714cfbd84dd38cf5c0f305ef6843 (table default.Table_2092a828aea549a38d1bd42248c8804c [id=cce6d993b26f4f9ca197d807107e00cb]) successfully deleted
W20250623 08:04:40.713615 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet c6a97b34e386443fa19a9601621b5017 because the tablet was not found. No further retry: Not found: Tablet not found: c6a97b34e386443fa19a9601621b5017
W20250623 08:04:40.714118 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet ea702abb31fb41498ab61bbb05b1f7e2 because the tablet was not found. No further retry: Not found: Tablet not found: ea702abb31fb41498ab61bbb05b1f7e2
W20250623 08:04:40.714668 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 2ecf4af9e476495683bd63cac20d4672 because the tablet was not found. No further retry: Not found: Tablet not found: 2ecf4af9e476495683bd63cac20d4672
I20250623 08:04:40.715499 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
W20250623 08:04:40.715601 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 4cb66c7a273a412db3ae77af9e30d036 because the tablet was not found. No further retry: Not found: Tablet not found: 4cb66c7a273a412db3ae77af9e30d036
W20250623 08:04:40.716039 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 279624b8bb834b95a4f08045968ebfb4 because the tablet was not found. No further retry: Not found: Tablet not found: 279624b8bb834b95a4f08045968ebfb4
I20250623 08:04:40.716296 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:40.716629 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]). Expected version 1 got 0
I20250623 08:04:40.717058 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:40.717505 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:40.717880 32057 catalog_manager.cc:5582] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:40.718513 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:40.719097 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 5 got 0
I20250623 08:04:40.719785 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 04fde7d1c1744c0597e43086ca4c9541 (DEFAULT_TABLE table=default.Table_06b731510ae743608f1fbc4d250acf14 [id=7fbe15151173464ea0ee480720c58b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:40.720511 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:40.721323 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:40.725133 339 tablet_bootstrap.cc:492] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:40.725360 339 ts_tablet_manager.cc:1397] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.255s user 0.087s sys 0.058s
I20250623 08:04:40.726133 339 raft_consensus.cc:357] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.726298 339 raft_consensus.cc:383] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.726364 339 raft_consensus.cc:738] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.726591 339 consensus_queue.cc:260] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.727152 339 ts_tablet_manager.cc:1428] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:40.728077 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fde7d1c1744c0597e43086ca4c9541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:40.724319 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_b91721eeaabb4377b4d62483ad065bee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:40.728789 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b91721eeaabb4377b4d62483ad065bee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:40.724325 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_1802c9f70d0f45868e11e3390504fade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:40.730708 339 tablet_bootstrap.cc:492] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:40.731206 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1802c9f70d0f45868e11e3390504fade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:40.732995 339 tablet_bootstrap.cc:654] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.734071 32057 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:40.729043 1627 consensus_queue.cc:1035] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.734287 979 log.cc:1199] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/ecb422550efd40cf82bc92fca5c25eda
I20250623 08:04:40.707072 343 tablet_bootstrap.cc:492] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:40.714155 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73453eb1a0394cfe8d49adcc71d4f3e9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.734539 32656 raft_consensus.cc:2466] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:40.718804 32589 leader_election.cc:304] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.735313 32589 leader_election.cc:304] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:40.735733 32589 leader_election.cc:343] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.735838 32589 leader_election.cc:304] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.735762 32057 catalog_manager.cc:5582] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:40.736296 1771 raft_consensus.cc:2747] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.736582 1771 raft_consensus.cc:2802] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:40.736642 1771 raft_consensus.cc:491] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.736708 1771 raft_consensus.cc:3058] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.738917 1863 raft_consensus.cc:2747] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.739535 1805 tablet.cc:1722] T 736eb56dae194df2af90bf33d3294a4e P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250623 08:04:40.740640 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_eb0f697365cc4f2b8bbeb5b3de47f1dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:40.741199 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb0f697365cc4f2b8bbeb5b3de47f1dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:40.743973 32050 master_service.cc:948] ReplaceTablet: received request to replace tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:40.743934 1771 raft_consensus.cc:513] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.744446 1771 leader_election.cc:290] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.746136 981 ts_tablet_manager.cc:1918] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:40.750553 343 tablet_bootstrap.cc:654] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.729245 32453 leader_election.cc:304] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:40.754288 1627 raft_consensus.cc:2762] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250623 08:04:40.761418 1771 raft_consensus.cc:491] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.761570 1771 raft_consensus.cc:513] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.762145 1771 leader_election.cc:290] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.762291 1359 consensus_queue.cc:1035] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:40.762882 979 ts_tablet_manager.cc:1939] T ecb422550efd40cf82bc92fca5c25eda P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:40.763653 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 329) took 1538 ms. Trace:
I20250623 08:04:40.763746 979 rpcz_store.cc:276] 0623 08:04:39.225457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.226250 (+ 793us) service_pool.cc:225] Handling call
0623 08:04:40.450075 (+1223825us) ts_tablet_manager.cc:1145] Deleting tablet ecb422550efd40cf82bc92fca5c25eda
0623 08:04:40.450083 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.622209 (+172126us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.700030 (+ 77821us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.763632 (+ 63602us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223706}
I20250623 08:04:40.764091 979 tablet_replica.cc:331] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:40.764261 979 raft_consensus.cc:2241] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.764238 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet ecb422550efd40cf82bc92fca5c25eda (table default.Table_68acbd516e8f4990a2f197ac131bc6ba [id=5545b396fc5746469e3a8908b0178a4c]) successfully deleted
I20250623 08:04:40.764411 979 raft_consensus.cc:2270] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.764850 979 ts_tablet_manager.cc:1905] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.765621 342 tablet_bootstrap.cc:492] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:40.765764 342 ts_tablet_manager.cc:1397] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.152s user 0.005s sys 0.001s
I20250623 08:04:40.766378 32453 leader_election.cc:304] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:40.766465 342 raft_consensus.cc:357] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.766606 342 raft_consensus.cc:383] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.766646 342 raft_consensus.cc:738] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.766884 1359 raft_consensus.cc:2802] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:40.766954 1359 raft_consensus.cc:491] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.767010 1359 raft_consensus.cc:3058] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.769383 342 consensus_queue.cc:260] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.769730 1359 raft_consensus.cc:513] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.769977 342 ts_tablet_manager.cc:1428] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250623 08:04:40.770222 342 tablet_bootstrap.cc:492] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:40.770300 1359 leader_election.cc:290] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.771828 342 tablet_bootstrap.cc:654] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:40.773936 32453 leader_election.cc:343] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.774094 32453 leader_election.cc:304] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.774320 1359 raft_consensus.cc:2747] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.775043 1809 tablet.cc:1722] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.781064 32054 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 388) took 1024 ms. Trace:
I20250623 08:04:40.781606 32054 rpcz_store.cc:276] 0623 08:04:39.756416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.740459 (+984043us) service_pool.cc:225] Handling call
0623 08:04:40.741291 (+ 832us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:40.741744 (+ 453us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:40.780329 (+ 38585us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:40.780833 (+ 504us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:40.780992 (+ 159us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:40.754619 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:40.754724 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:40.754729 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:40.754732 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:40.754847 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:40.754856 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:40.754861 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:40.754925 (+ 64us) tablet.cc:699] Row locks acquired
0623 08:04:40.754927 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:40.754953 (+ 26us) write_op.cc:270] Start()
0623 08:04:40.754976 (+ 23us) write_op.cc:276] Timestamp: P: 1750665880754898 usec, L: 0
0623 08:04:40.754979 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:40.755109 (+ 130us) log.cc:844] Serialized 433 byte log entry
0623 08:04:40.776111 (+ 21002us) op_driver.cc:464] REPLICATION: finished
0623 08:04:40.778914 (+ 2803us) write_op.cc:301] APPLY: starting
0623 08:04:40.778931 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:40.778974 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:40.779008 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:40.779111 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:40.779145 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:40.779153 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:40.779228 (+ 75us) log.cc:844] Serialized 29 byte log entry
0623 08:04:40.780001 (+ 773us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.780045 (+ 44us) write_op.cc:454] Released schema lock
0623 08:04:40.780055 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:40.780167 (+ 112us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.780174 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2738,"apply.run_cpu_time_us":555,"apply.run_wall_time_us":1243,"mutex_wait_us":12377,"num_ops":2,"prepare.queue_time_us":12477,"prepare.run_cpu_time_us":552,"prepare.run_wall_time_us":552,"replication_time_us":21102}]]}
I20250623 08:04:40.784960 32054 master_service.cc:948] ReplaceTablet: received request to replace tablet d5cd316e6f4447be958611ab07ed31fc from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:40.788944 1359 consensus_queue.cc:1035] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:40.790356 32453 leader_election.cc:343] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.792737 32050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3
I20250623 08:04:40.793313 32050 catalog_manager.cc:6652] ReplaceTablet: tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 deleted and replaced by tablet a79238b6f6e74ae89c52eefb95190264
I20250623 08:04:40.793434 32050 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36116 (request call id 389) took 1013 ms. Trace:
I20250623 08:04:40.793679 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 with delete_type TABLET_DATA_DELETED (replaced by tablet a79238b6f6e74ae89c52eefb95190264) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:40.793533 32050 rpcz_store.cc:276] 0623 08:04:39.780232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.743897 (+963665us) service_pool.cc:225] Handling call
0623 08:04:40.746812 (+ 2915us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d080605b 00007f19c764f5d7 00007f19c764f305 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:40.793404 (+ 46592us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:40.755203 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:40.755299 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:40.755305 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:40.755307 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:40.755389 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:40.755396 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:40.755400 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:40.755453 (+ 53us) tablet.cc:699] Row locks acquired
0623 08:04:40.755455 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:40.755481 (+ 26us) write_op.cc:270] Start()
0623 08:04:40.755521 (+ 40us) write_op.cc:276] Timestamp: P: 1750665880755425 usec, L: 0
0623 08:04:40.755526 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:40.755662 (+ 136us) log.cc:844] Serialized 622 byte log entry
0623 08:04:40.776180 (+ 20518us) op_driver.cc:464] REPLICATION: finished
0623 08:04:40.780071 (+ 3891us) write_op.cc:301] APPLY: starting
0623 08:04:40.780085 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:40.780118 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:40.780139 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:40.780254 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:40.780306 (+ 52us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:40.780313 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:40.780375 (+ 62us) log.cc:844] Serialized 29 byte log entry
0623 08:04:40.780442 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.780459 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:40.780464 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:40.780530 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.780534 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":5848960,"child_traces":[["op",{"apply.queue_time_us":3833,"apply.run_cpu_time_us":458,"apply.run_wall_time_us":458,"mutex_wait_us":6939,"num_ops":2,"prepare.queue_time_us":7562,"prepare.run_cpu_time_us":519,"prepare.run_wall_time_us":518,"replication_time_us":20635}]]}
I20250623 08:04:40.794313 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_fb578541a1094c0f90ba5281797a457d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:40.792959 1359 consensus_queue.cc:1035] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20250623 08:04:40.795785 32453 consensus_peers.cc:487] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:40.794248 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 with delete_type TABLET_DATA_DELETED (replaced by tablet a79238b6f6e74ae89c52eefb95190264) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:40.797081 981 log.cc:1199] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/26f4a1b472e848ccb9a1ec71c1dd3516
I20250623 08:04:40.797549 981 ts_tablet_manager.cc:1939] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:40.797863 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "789d1c20aa72480ab71123c799bbde4c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:40.798020 32516 raft_consensus.cc:3058] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.798321 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 223) took 4758 ms. Trace:
I20250623 08:04:40.798393 981 rpcz_store.cc:276] 0623 08:04:36.040200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.040523 (+ 323us) service_pool.cc:225] Handling call
0623 08:04:40.686325 (+4645802us) ts_tablet_manager.cc:1145] Deleting tablet 26f4a1b472e848ccb9a1ec71c1dd3516
0623 08:04:40.686339 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.734952 (+ 48613us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.746014 (+ 11062us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.798303 (+ 52289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4645582}
I20250623 08:04:40.798885 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 26f4a1b472e848ccb9a1ec71c1dd3516 (table default.Table_61ccd9db16cb4618874bff73413fd485 [id=8d6f6785bcf24b719419015368a05f68]) successfully deleted
I20250623 08:04:40.799187 981 tablet_replica.cc:331] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
W20250623 08:04:40.799232 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb578541a1094c0f90ba5281797a457d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:40.799357 981 raft_consensus.cc:2241] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:40.799528 981 raft_consensus.cc:2270] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:40.800051 981 ts_tablet_manager.cc:1905] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:40.800771 32516 raft_consensus.cc:2466] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:40.801894 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9049e3d9b39f420bb51221808409fc54" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.802063 32516 raft_consensus.cc:2466] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:40.803959 343 tablet_bootstrap.cc:492] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:40.804124 343 ts_tablet_manager.cc:1397] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.001s
I20250623 08:04:40.804472 1715 mvcc.cc:204] Tried to move back new op lower bound from 7170727422154145792 to 7170727396518666240. Current Snapshot: MvccSnapshot[applied={T|T < 7170727422154145792 or (T in {7170727422154145792})}]
I20250623 08:04:40.805389 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73453eb1a0394cfe8d49adcc71d4f3e9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:40.805553 32642 raft_consensus.cc:3058] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.805649 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 with delete_type TABLET_DATA_DELETED (replaced by tablet a79238b6f6e74ae89c52eefb95190264) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:40.808213 32642 raft_consensus.cc:2466] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:40.810312 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "73453eb1a0394cfe8d49adcc71d4f3e9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.812849 1810 tablet.cc:1722] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:40.813105 343 raft_consensus.cc:357] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:40.813301 343 raft_consensus.cc:383] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.813365 343 raft_consensus.cc:738] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.813585 343 consensus_queue.cc:260] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:40.814203 343 ts_tablet_manager.cc:1428] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250623 08:04:40.814474 343 tablet_bootstrap.cc:492] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:40.819152 343 tablet_bootstrap.cc:654] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.825923 32054 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet d5cd316e6f4447be958611ab07ed31fc
I20250623 08:04:40.826231 32371 raft_consensus.cc:3058] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.826457 32054 catalog_manager.cc:6652] ReplaceTablet: tablet d5cd316e6f4447be958611ab07ed31fc deleted and replaced by tablet c50655c06cd94c67a49513c39728eaa9
I20250623 08:04:40.830472 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet d5cd316e6f4447be958611ab07ed31fc with delete_type TABLET_DATA_DELETED (replaced by tablet c50655c06cd94c67a49513c39728eaa9) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:40.832090 1843 raft_consensus.cc:491] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:40.832273 1843 raft_consensus.cc:513] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:40.833027 1843 leader_election.cc:290] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.833372 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet d5cd316e6f4447be958611ab07ed31fc with delete_type TABLET_DATA_DELETED (replaced by tablet c50655c06cd94c67a49513c39728eaa9) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:40.834573 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b81263a9ee9d4da1ac721c3b2bd21248" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:40.838840 32314 consensus_peers.cc:487] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba3d2fa01d574dd992dc904ac024ba3a. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:40.839265 32314 leader_election.cc:343] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: b81263a9ee9d4da1ac721c3b2bd21248
I20250623 08:04:40.839550 32314 leader_election.cc:304] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:40.839941 1843 raft_consensus.cc:2747] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:40.840211 1627 raft_consensus.cc:491] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.843430 1627 raft_consensus.cc:513] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.843670 1843 raft_consensus.cc:491] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:40.843827 1843 raft_consensus.cc:513] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.844630 1843 leader_election.cc:290] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:40.845650 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861e45cf546487d9812caa06e2704eb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.845929 32516 raft_consensus.cc:2391] T 0861e45cf546487d9812caa06e2704eb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:40.846016 1627 leader_election.cc:290] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.846652 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0861e45cf546487d9812caa06e2704eb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.847831 32314 leader_election.cc:304] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:40.848219 1843 raft_consensus.cc:3058] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.852559 1843 raft_consensus.cc:2747] T 0861e45cf546487d9812caa06e2704eb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:40.854144 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37b38cb54d6c45308358f556a692eabe" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.856173 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37b38cb54d6c45308358f556a692eabe" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:40.856742 32642 raft_consensus.cc:2466] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:40.841656 952 ts_tablet_manager.cc:1918] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:40.867352 952 log.cc:1199] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/965f1f0a107b4653b6aac01e0d8e259a
I20250623 08:04:40.868350 952 ts_tablet_manager.cc:1939] T 965f1f0a107b4653b6aac01e0d8e259a P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:40.868624 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet d5cd316e6f4447be958611ab07ed31fc with delete_type TABLET_DATA_DELETED (replaced by tablet c50655c06cd94c67a49513c39728eaa9) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:40.878918 32453 leader_election.cc:343] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.886371 343 tablet_bootstrap.cc:492] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:40.886541 343 ts_tablet_manager.cc:1397] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.072s user 0.027s sys 0.027s
I20250623 08:04:40.887316 343 raft_consensus.cc:357] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:40.887486 343 raft_consensus.cc:383] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.887542 343 raft_consensus.cc:738] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.888025 343 consensus_queue.cc:260] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:40.889334 343 ts_tablet_manager.cc:1428] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250623 08:04:40.888875 32453 leader_election.cc:343] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.889951 343 tablet_bootstrap.cc:492] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:40.896459 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 185) took 5594 ms. Trace:
I20250623 08:04:40.896555 952 rpcz_store.cc:276] 0623 08:04:35.301668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.301709 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:40.579233 (+5277524us) ts_tablet_manager.cc:1145] Deleting tablet 965f1f0a107b4653b6aac01e0d8e259a
0623 08:04:40.579242 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.753011 (+173769us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.841514 (+ 88503us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.896434 (+ 54920us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14702,"tablet-delete.queue_time_us":5277328}
I20250623 08:04:40.896983 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 182) took 5599 ms. Trace:
I20250623 08:04:40.897069 952 rpcz_store.cc:276] 0623 08:04:35.297984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.299470 (+ 1486us) service_pool.cc:225] Handling call
0623 08:04:40.896880 (+5597410us) ts_tablet_manager.cc:1145] Deleting tablet 2060a643d0da47efa868f52bdb331b1f
0623 08:04:40.896893 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.896966 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5592257}
I20250623 08:04:40.897327 952 tablet_replica.cc:331] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:40.921047 343 tablet_bootstrap.cc:654] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:40.923669 32453 leader_election.cc:304] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.924224 342 tablet_bootstrap.cc:492] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:40.924968 342 ts_tablet_manager.cc:1397] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.155s user 0.000s sys 0.005s
I20250623 08:04:40.925287 1823 raft_consensus.cc:2802] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:40.927039 1823 raft_consensus.cc:695] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:40.927285 1823 consensus_queue.cc:237] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:40.932122 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 195990us
I20250623 08:04:40.932986 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8de2b1105564fdab386442b1de4cedb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.937281 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37b38cb54d6c45308358f556a692eabe" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:40.944808 32453 leader_election.cc:304] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:40.945727 1626 raft_consensus.cc:2802] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:40.945834 1626 raft_consensus.cc:491] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:40.945911 1626 raft_consensus.cc:3058] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.948951 1626 raft_consensus.cc:513] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.949590 1626 leader_election.cc:290] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:40.949833 979 ts_tablet_manager.cc:1918] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:40.950066 342 raft_consensus.cc:357] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.950214 342 raft_consensus.cc:383] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.950263 342 raft_consensus.cc:738] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.950424 342 consensus_queue.cc:260] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.958513 342 ts_tablet_manager.cc:1428] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.033s user 0.000s sys 0.001s
I20250623 08:04:40.959939 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 965f1f0a107b4653b6aac01e0d8e259a (table default.Table_5f159a56c60144b99a6847ec6080623e [id=ec1f9e35e20c47fcbe2d4b80c7abeabc]) successfully deleted
W20250623 08:04:40.960260 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 2060a643d0da47efa868f52bdb331b1f because the tablet was not found. No further retry: Not found: Tablet not found: 2060a643d0da47efa868f52bdb331b1f
I20250623 08:04:40.962503 339 tablet_bootstrap.cc:492] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:40.962775 339 ts_tablet_manager.cc:1397] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.232s user 0.007s sys 0.000s
I20250623 08:04:40.963757 339 raft_consensus.cc:357] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.964066 339 raft_consensus.cc:383] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:40.964177 339 raft_consensus.cc:738] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:40.964437 339 consensus_queue.cc:260] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.964483 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37b38cb54d6c45308358f556a692eabe" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:40.965116 339 ts_tablet_manager.cc:1428] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:40.965417 339 tablet_bootstrap.cc:492] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:40.969977 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:40.978128 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8de2b1105564fdab386442b1de4cedb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:40.978322 32516 raft_consensus.cc:3058] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:40.981748 32516 raft_consensus.cc:2466] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:40.982100 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37b38cb54d6c45308358f556a692eabe" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:40.982339 32514 raft_consensus.cc:2391] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:40.983611 1809 tablet.cc:1722] T 736eb56dae194df2af90bf33d3294a4e P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20250623 08:04:40.984090 32453 leader_election.cc:343] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:40.992806 32054 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c76a280c52214058b9b4b2691090cc1f" } new_table_name: "default.Table_b764664d1bdc4441907690b6616f3570" modify_external_catalogs: true
I20250623 08:04:40.992873 32050 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fb2522ad8b664fdc9ec779b46b807ca1" } modify_external_catalogs: true
I20250623 08:04:40.993072 32050 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_fb2522ad8b664fdc9ec779b46b807ca1" } modify_external_catalogs: true
I20250623 08:04:41.000559 981 ts_tablet_manager.cc:1918] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.000783 981 log.cc:1199] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/2f62ec83c88e43438b963e740d2c7004
I20250623 08:04:41.001598 981 ts_tablet_manager.cc:1939] T 2f62ec83c88e43438b963e740d2c7004 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:41.002460 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 225) took 4846 ms. Trace:
I20250623 08:04:41.002599 981 rpcz_store.cc:276] 0623 08:04:36.156080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.156247 (+ 167us) service_pool.cc:225] Handling call
0623 08:04:40.799137 (+4642890us) ts_tablet_manager.cc:1145] Deleting tablet 2f62ec83c88e43438b963e740d2c7004
0623 08:04:40.799148 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.996297 (+197149us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.000456 (+ 4159us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.002442 (+ 1986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4642688}
I20250623 08:04:41.003154 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 2f62ec83c88e43438b963e740d2c7004 (table default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac [id=65c4e79e095b41b2b2a42e9858485423]) successfully deleted
I20250623 08:04:41.006417 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 227) took 4820 ms. Trace:
I20250623 08:04:41.006553 981 rpcz_store.cc:276] 0623 08:04:36.186251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.186300 (+ 49us) service_pool.cc:225] Handling call
0623 08:04:41.003582 (+4817282us) ts_tablet_manager.cc:1145] Deleting tablet 16331ba6a34942a5a1b4ff4ac7bac55e
0623 08:04:41.003592 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.006398 (+ 2806us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4816919}
W20250623 08:04:41.007148 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 16331ba6a34942a5a1b4ff4ac7bac55e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16331ba6a34942a5a1b4ff4ac7bac55e already in progress: creating tablet
I20250623 08:04:41.007262 1810 tablet.cc:1722] T 736eb56dae194df2af90bf33d3294a4e P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250623 08:04:41.015794 342 tablet_bootstrap.cc:492] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:40.953068 32589 leader_election.cc:304] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.016451 979 log.cc:1199] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/3249605158634f4eab5b9d61357c383f
W20250623 08:04:41.016639 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 63600.3us
I20250623 08:04:41.016884 32589 leader_election.cc:304] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.017159 342 tablet_bootstrap.cc:654] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.017356 32589 leader_election.cc:304] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:40.964956 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37b38cb54d6c45308358f556a692eabe" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:41.017805 32646 raft_consensus.cc:3058] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.018031 1877 raft_consensus.cc:2802] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.018306 1854 raft_consensus.cc:2802] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:41.018373 1854 raft_consensus.cc:491] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:41.018440 1854 raft_consensus.cc:3058] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.020414 32646 raft_consensus.cc:2466] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:41.021549 1854 raft_consensus.cc:513] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.022073 1854 leader_election.cc:290] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.022336 1854 raft_consensus.cc:2802] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.022459 1854 raft_consensus.cc:695] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:41.034304 32453 leader_election.cc:304] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.035135 1359 raft_consensus.cc:2802] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.035347 1359 raft_consensus.cc:695] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:41.035574 1359 consensus_queue.cc:237] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:40.977504 343 tablet_bootstrap.cc:492] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:41.014382 1863 raft_consensus.cc:491] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.040500 343 ts_tablet_manager.cc:1397] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.005s
I20250623 08:04:41.040506 1863 raft_consensus.cc:513] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.041266 1863 leader_election.cc:290] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.041373 343 raft_consensus.cc:357] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.041586 343 raft_consensus.cc:383] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.041647 343 raft_consensus.cc:738] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.041867 343 consensus_queue.cc:260] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.042467 343 ts_tablet_manager.cc:1428] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:41.051949 952 raft_consensus.cc:2241] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:41.052323 952 raft_consensus.cc:2270] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.052888 952 ts_tablet_manager.cc:1905] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.056974 343 tablet_bootstrap.cc:492] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:41.022687 1854 consensus_queue.cc:237] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.058372 343 tablet_bootstrap.cc:654] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.076159 342 tablet_bootstrap.cc:492] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.076300 342 ts_tablet_manager.cc:1397] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.007s
I20250623 08:04:41.076954 342 raft_consensus.cc:357] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.077122 342 raft_consensus.cc:383] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.077175 342 raft_consensus.cc:738] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.077458 342 consensus_queue.cc:260] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.078580 342 ts_tablet_manager.cc:1428] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:41.080376 342 tablet_bootstrap.cc:492] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.080685 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:41.080930 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:41.082087 342 tablet_bootstrap.cc:654] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.084415 1863 raft_consensus.cc:491] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.084589 1863 raft_consensus.cc:513] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.085944 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:41.086385 952 ts_tablet_manager.cc:1918] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.088817 1863 leader_election.cc:290] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.090034 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:41.094353 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a8de2b1105564fdab386442b1de4cedb (table default.Table_0ac52a7e208745e980f3a49c46b3cdb8 [id=1f77f0d0a7fc4af9986e39dfee69bfee]). Expected version 5 got 0
I20250623 08:04:41.053263 1843 raft_consensus.cc:491] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.099354 1843 raft_consensus.cc:513] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.012054 339 tablet_bootstrap.cc:654] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.093998 1359 consensus_queue.cc:1035] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.100560 952 log.cc:1199] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a414144276f6484ea4690204752e2142
I20250623 08:04:41.101307 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b405f87bb1e4d0686a1f627835d901f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:41.101596 952 ts_tablet_manager.cc:1939] T a414144276f6484ea4690204752e2142 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:41.102242 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2b405f87bb1e4d0686a1f627835d901f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.102463 32519 raft_consensus.cc:2391] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:41.102926 32314 leader_election.cc:304] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:41.103807 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 186) took 5801 ms. Trace:
I20250623 08:04:41.104424 952 rpcz_store.cc:276] 0623 08:04:35.302080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.302739 (+ 659us) service_pool.cc:225] Handling call
0623 08:04:40.897281 (+5594542us) ts_tablet_manager.cc:1145] Deleting tablet a414144276f6484ea4690204752e2142
0623 08:04:40.897291 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.065116 (+167825us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.086272 (+ 21156us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.103791 (+ 17519us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5592608}
I20250623 08:04:41.107138 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 187) took 5803 ms. Trace:
I20250623 08:04:41.107231 1843 leader_election.cc:290] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.107458 952 rpcz_store.cc:276] 0623 08:04:35.303019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.303065 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:41.106659 (+5803594us) ts_tablet_manager.cc:1145] Deleting tablet 79731106915d4cebb9dcc3c10386ffa9
0623 08:04:41.106669 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.106733 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5801936}
I20250623 08:04:41.108372 952 tablet_replica.cc:331] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:41.108925 952 raft_consensus.cc:2241] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:41.110070 952 pending_rounds.cc:70] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:41.110787 952 pending_rounds.cc:77] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727440241176576 op_type: NO_OP noop_request { }
I20250623 08:04:41.112035 339 tablet_bootstrap.cc:492] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.112156 339 ts_tablet_manager.cc:1397] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.147s user 0.002s sys 0.005s
I20250623 08:04:41.112871 339 raft_consensus.cc:357] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.113019 339 raft_consensus.cc:383] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.113062 339 raft_consensus.cc:738] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.113243 339 consensus_queue.cc:260] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.113692 339 ts_tablet_manager.cc:1428] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:41.113931 339 tablet_bootstrap.cc:492] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.116250 339 tablet_bootstrap.cc:654] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.117545 952 raft_consensus.cc:2887] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:41.117913 952 raft_consensus.cc:2270] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.118042 1908 raft_consensus.cc:3058] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.118810 952 ts_tablet_manager.cc:1905] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.121618 1908 raft_consensus.cc:2747] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.017364 979 ts_tablet_manager.cc:1939] T 3249605158634f4eab5b9d61357c383f P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:41.124856 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 330) took 1899 ms. Trace:
I20250623 08:04:41.125002 979 rpcz_store.cc:276] 0623 08:04:39.225551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.226397 (+ 846us) service_pool.cc:225] Handling call
0623 08:04:40.764035 (+1537638us) ts_tablet_manager.cc:1145] Deleting tablet 3249605158634f4eab5b9d61357c383f
0623 08:04:40.764048 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:40.946484 (+182436us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:40.949736 (+ 3252us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.124836 (+175100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1537449}
I20250623 08:04:41.106210 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet a414144276f6484ea4690204752e2142 (table default.Table_a4ca66fa08cc441ab5571f9968f30914 [id=c3c5e1869ba745b9b2ab29742806f241]) successfully deleted
I20250623 08:04:41.125530 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 3249605158634f4eab5b9d61357c383f (table default.Table_79be3d7247284828a82f02d96faa60cc [id=c618740b8cb040bcb78c452b3d747391]) successfully deleted
I20250623 08:04:41.125645 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:41.125617 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 332) took 1899 ms. Trace:
I20250623 08:04:41.125782 979 rpcz_store.cc:276] 0623 08:04:39.225726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.226622 (+ 896us) service_pool.cc:225] Handling call
0623 08:04:41.125500 (+1898878us) ts_tablet_manager.cc:1145] Deleting tablet f4a0b9402a264800a1ea39707c10da0c
0623 08:04:41.125511 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.125604 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898760}
I20250623 08:04:41.126232 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 325) took 1901 ms. Trace:
I20250623 08:04:41.126361 979 rpcz_store.cc:276] 0623 08:04:39.225087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.226846 (+ 1759us) service_pool.cc:225] Handling call
0623 08:04:41.126165 (+1899319us) ts_tablet_manager.cc:1145] Deleting tablet 136e482995a24c5daec55a701f3091fa
0623 08:04:41.126174 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.126220 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1899174}
I20250623 08:04:41.126770 342 tablet_bootstrap.cc:492] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.126894 342 ts_tablet_manager.cc:1397] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.003s
W20250623 08:04:41.125839 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 79731106915d4cebb9dcc3c10386ffa9 because the tablet was not found. No further retry: Not found: Tablet not found: 79731106915d4cebb9dcc3c10386ffa9
W20250623 08:04:41.129091 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f4a0b9402a264800a1ea39707c10da0c because the tablet was not found. No further retry: Not found: Tablet not found: f4a0b9402a264800a1ea39707c10da0c
W20250623 08:04:41.129350 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 136e482995a24c5daec55a701f3091fa because the tablet was not found. No further retry: Not found: Tablet not found: 136e482995a24c5daec55a701f3091fa
I20250623 08:04:41.129477 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:41.130174 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:41.135562 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:41.137126 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:41.138322 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517]). Expected version 5 got 0
I20250623 08:04:41.140210 32053 catalog_manager.cc:5582] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:41.143128 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:41.145202 342 raft_consensus.cc:357] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.145391 342 raft_consensus.cc:383] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.145460 342 raft_consensus.cc:738] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.146368 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:41.146953 342 consensus_queue.cc:260] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.147392 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:41.147758 342 ts_tablet_manager.cc:1428] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250623 08:04:41.148026 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:41.007704 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 229) took 4592 ms. Trace:
I20250623 08:04:41.148123 981 rpcz_store.cc:276] 0623 08:04:36.415161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.418797 (+ 3636us) service_pool.cc:225] Handling call
0623 08:04:41.007587 (+4588790us) ts_tablet_manager.cc:1145] Deleting tablet b84b8b8713774ab4896842feb9fb075e
0623 08:04:41.007597 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.007689 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4588605}
I20250623 08:04:41.148633 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:41.148794 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 230) took 4733 ms. Trace:
I20250623 08:04:41.148941 981 rpcz_store.cc:276] 0623 08:04:36.415255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.419015 (+ 3760us) service_pool.cc:225] Handling call
0623 08:04:41.148626 (+4729611us) ts_tablet_manager.cc:1145] Deleting tablet 9bfa787f828b411ab68d81bb896f7fb6
0623 08:04:41.148639 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.148776 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4729442}
I20250623 08:04:41.149293 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:41.149341 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 238) took 4597 ms. Trace:
I20250623 08:04:41.149408 981 rpcz_store.cc:276] 0623 08:04:36.551828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.570808 (+ 18980us) service_pool.cc:225] Handling call
0623 08:04:41.149229 (+4578421us) ts_tablet_manager.cc:1145] Deleting tablet f5f58f55070344ff98a3808f70e0c755
0623 08:04:41.149239 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.149327 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4578254}
I20250623 08:04:41.150038 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:41.150123 981 tablet_replica.cc:331] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:41.150332 981 raft_consensus.cc:2241] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:41.150528 981 raft_consensus.cc:2270] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.150763 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:41.151140 981 ts_tablet_manager.cc:1905] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.151706 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:41.153249 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:41.154453 1359 consensus_queue.cc:1035] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.155223 342 tablet_bootstrap.cc:492] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.155840 979 tablet_replica.cc:331] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:41.156473 979 raft_consensus.cc:2241] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:41.154037 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet b84b8b8713774ab4896842feb9fb075e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b84b8b8713774ab4896842feb9fb075e already in progress: creating tablet
W20250623 08:04:41.156895 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 9bfa787f828b411ab68d81bb896f7fb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bfa787f828b411ab68d81bb896f7fb6 already in progress: creating tablet
I20250623 08:04:41.156947 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
W20250623 08:04:41.157181 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet f5f58f55070344ff98a3808f70e0c755 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5f58f55070344ff98a3808f70e0c755 already in progress: creating tablet
I20250623 08:04:41.157557 979 raft_consensus.cc:2270] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.157199 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:41.159227 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:41.159415 342 tablet_bootstrap.cc:654] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.160609 1784 catalog_manager.cc:1261] Loaded cluster ID: bde910f56f35432e94ef19d006edb93c
I20250623 08:04:41.160728 1784 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: loading cluster ID for follower catalog manager: success
I20250623 08:04:41.160959 339 tablet_bootstrap.cc:492] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.161108 339 ts_tablet_manager.cc:1397] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.002s
I20250623 08:04:41.161849 339 raft_consensus.cc:357] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.162051 339 raft_consensus.cc:383] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.162104 339 raft_consensus.cc:738] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.162325 339 consensus_queue.cc:260] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.162953 339 ts_tablet_manager.cc:1428] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:41.163250 339 tablet_bootstrap.cc:492] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.163604 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_217651efa5dc4571b2a47cd3196c9e57 [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 1 got 0
I20250623 08:04:41.164670 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1c529e826fa84a919c018887620e4023 (table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]). Expected version 1 got 0
I20250623 08:04:41.165000 339 tablet_bootstrap.cc:654] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.165762 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:41.167865 1877 raft_consensus.cc:695] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:41.168385 1877 consensus_queue.cc:237] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:41.169581 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 151842us
I20250623 08:04:41.172013 952 ts_tablet_manager.cc:1918] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.172688 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9049e3d9b39f420bb51221808409fc54" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:41.173870 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:41.173976 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5af1dea3c6ca4f498ce6c039d72a14b7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.176659 979 ts_tablet_manager.cc:1905] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.178733 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:41.180147 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:41.182547 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:41.182842 952 log.cc:1199] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/d0b2af698b6e4ed0ae66a9bb32be7b31
I20250623 08:04:41.173449 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9049e3d9b39f420bb51221808409fc54" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:41.183163 32519 raft_consensus.cc:3058] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.186245 343 tablet_bootstrap.cc:492] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:41.183513 952 ts_tablet_manager.cc:1939] T d0b2af698b6e4ed0ae66a9bb32be7b31 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:41.186381 343 ts_tablet_manager.cc:1397] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.144s user 0.001s sys 0.014s
I20250623 08:04:41.178324 342 tablet_bootstrap.cc:492] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.178485 1823 raft_consensus.cc:491] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:41.185820 32519 raft_consensus.cc:2466] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:41.174706 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5af1dea3c6ca4f498ce6c039d72a14b7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.186594 1823 raft_consensus.cc:513] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.186838 32518 raft_consensus.cc:2466] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:41.187240 1823 leader_election.cc:290] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.187161 343 raft_consensus.cc:357] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.187350 1823 raft_consensus.cc:491] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.187436 1823 raft_consensus.cc:513] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.187901 1823 leader_election.cc:290] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:41.188045 32453 leader_election.cc:343] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 2f62ec83c88e43438b963e740d2c7004
W20250623 08:04:41.188199 32453 leader_election.cc:343] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 2f62ec83c88e43438b963e740d2c7004
I20250623 08:04:41.188236 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f83f68ebda7e4826bbb17278cd4ac5a8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.188282 32453 leader_election.cc:304] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.188251 1908 raft_consensus.cc:491] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.188470 1823 raft_consensus.cc:2747] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:41.188620 1908 raft_consensus.cc:513] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.188975 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b5233522f43c7aed3c9770adb5c03" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:41.189142 32646 raft_consensus.cc:2466] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:41.189399 1908 leader_election.cc:290] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.189568 32453 leader_election.cc:304] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:41.189677 1789 raft_consensus.cc:491] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.189765 1823 raft_consensus.cc:2802] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:41.189831 1823 raft_consensus.cc:491] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:41.189790 1789 raft_consensus.cc:513] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.189889 1823 raft_consensus.cc:3058] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.190280 1789 leader_election.cc:290] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.187315 343 raft_consensus.cc:383] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.190380 1789 raft_consensus.cc:491] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.190407 343 raft_consensus.cc:738] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.190459 1789 raft_consensus.cc:513] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.190639 343 consensus_queue.cc:260] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.190919 1789 leader_election.cc:290] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.191090 343 ts_tablet_manager.cc:1428] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:41.191334 343 tablet_bootstrap.cc:492] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:41.191358 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38eecf4a94d4536b98d4f6e037a0094" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:41.191959 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a38eecf4a94d4536b98d4f6e037a0094" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.192133 32518 raft_consensus.cc:2391] T a38eecf4a94d4536b98d4f6e037a0094 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:41.192456 1823 raft_consensus.cc:513] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.193007 343 tablet_bootstrap.cc:654] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.192999 1823 leader_election.cc:290] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.193001 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b5233522f43c7aed3c9770adb5c03" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.193380 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f83f68ebda7e4826bbb17278cd4ac5a8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.193562 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b5233522f43c7aed3c9770adb5c03" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:41.193728 32646 raft_consensus.cc:3058] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.194095 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b5233522f43c7aed3c9770adb5c03" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:41.194165 32314 leader_election.cc:304] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
W20250623 08:04:41.194420 32589 leader_election.cc:343] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.194593 1908 raft_consensus.cc:3058] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:41.194787 32589 leader_election.cc:343] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.194953 32589 leader_election.cc:304] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.195250 1789 raft_consensus.cc:2747] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.196349 32646 raft_consensus.cc:2466] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:41.196841 32589 leader_election.cc:343] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.196900 32453 leader_election.cc:304] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:41.197080 32589 leader_election.cc:343] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:41.197176 32453 leader_election.cc:343] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.197275 1908 raft_consensus.cc:2747] T a38eecf4a94d4536b98d4f6e037a0094 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.197373 1789 consensus_queue.cc:1035] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:41.197489 32453 leader_election.cc:343] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.185408 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:41.197819 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:41.197937 1627 raft_consensus.cc:2802] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.198050 1627 raft_consensus.cc:695] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:41.198118 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
W20250623 08:04:41.198086 32589 leader_election.cc:343] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.198279 1627 consensus_queue.cc:237] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.198467 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:41.198666 32053 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:41.200569 342 ts_tablet_manager.cc:1397] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250623 08:04:41.201334 342 raft_consensus.cc:357] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.201493 342 raft_consensus.cc:383] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.201563 342 raft_consensus.cc:738] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.201843 342 consensus_queue.cc:260] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.202581 342 ts_tablet_manager.cc:1428] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:41.202920 339 tablet_bootstrap.cc:492] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.203054 339 ts_tablet_manager.cc:1397] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.001s
I20250623 08:04:41.203881 32059 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet aa398354a9e44633857a75c0bbba08b3
I20250623 08:04:41.204463 32059 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 378) took 1695 ms. Trace:
I20250623 08:04:41.204574 32059 rpcz_store.cc:276] 0623 08:04:39.509436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.738001 (+228565us) service_pool.cc:225] Handling call
0623 08:04:39.763730 (+ 25729us) spinlock_profiling.cc:243] Waited 25.5 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fa306 00007f19c7644956 00007f19c7644685 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:39.764099 (+ 369us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:39.764219 (+ 120us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:39.764285 (+ 66us) catalog_manager.cc:2799] Locking tablets
0623 08:04:39.764482 (+ 197us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:39.860926 (+ 96444us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:39.861014 (+ 88us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:41.203802 (+1342788us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:41.204409 (+ 607us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:39.782276 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:39.782440 (+ 164us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:39.782446 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:39.782449 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:39.782566 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:39.782578 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:39.782582 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:39.782696 (+ 114us) tablet.cc:699] Row locks acquired
0623 08:04:39.782698 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:39.782735 (+ 37us) write_op.cc:270] Start()
0623 08:04:39.782804 (+ 69us) write_op.cc:276] Timestamp: P: 1750665879782671 usec, L: 0
0623 08:04:39.782808 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:39.783001 (+ 193us) log.cc:844] Serialized 759 byte log entry
0623 08:04:39.855331 (+ 72330us) op_driver.cc:464] REPLICATION: finished
0623 08:04:39.856698 (+ 1367us) write_op.cc:301] APPLY: starting
0623 08:04:39.856707 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:39.856740 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:39.856761 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:39.856817 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:39.856840 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:39.856847 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:39.856891 (+ 44us) log.cc:844] Serialized 29 byte log entry
0623 08:04:39.856924 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.856936 (+ 12us) write_op.cc:454] Released schema lock
0623 08:04:39.856941 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:39.856990 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.856994 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":58589056,"child_traces":[["op",{"apply.queue_time_us":1334,"apply.run_cpu_time_us":288,"apply.run_wall_time_us":288,"num_ops":2,"prepare.queue_time_us":17088,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":2255,"replication_time_us":72507}]]}
I20250623 08:04:41.205626 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 188) took 5901 ms. Trace:
I20250623 08:04:41.205709 952 rpcz_store.cc:276] 0623 08:04:35.303915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.303964 (+ 49us) service_pool.cc:225] Handling call
0623 08:04:41.108326 (+5804362us) ts_tablet_manager.cc:1145] Deleting tablet d0b2af698b6e4ed0ae66a9bb32be7b31
0623 08:04:41.108334 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.149432 (+ 41098us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.171901 (+ 22469us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.205609 (+ 33708us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5803564}
I20250623 08:04:41.206120 342 tablet_bootstrap.cc:492] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.207270 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet d0b2af698b6e4ed0ae66a9bb32be7b31 (table default.Table_d3f72251faee462c9c301b8aff464bb5 [id=b7671dfa27be46bb8b811b12016434e2]) successfully deleted
I20250623 08:04:41.207508 342 tablet_bootstrap.cc:654] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.207760 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet aa398354a9e44633857a75c0bbba08b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.208277 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 189) took 5904 ms. Trace:
I20250623 08:04:41.208381 952 rpcz_store.cc:276] 0623 08:04:35.304213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.304257 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:41.208184 (+5903927us) ts_tablet_manager.cc:1145] Deleting tablet 2795cb14e6db4116a4c9ee539bd4e82d
0623 08:04:41.208192 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.208261 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5903387}
I20250623 08:04:41.208773 952 tablet_replica.cc:331] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:41.208999 952 raft_consensus.cc:2241] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:41.209365 952 raft_consensus.cc:2270] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.209936 952 ts_tablet_manager.cc:1905] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.211295 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet aa398354a9e44633857a75c0bbba08b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:41.211400 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 2795cb14e6db4116a4c9ee539bd4e82d because the tablet was not found. No further retry: Not found: Tablet not found: 2795cb14e6db4116a4c9ee539bd4e82d
I20250623 08:04:41.211773 1908 raft_consensus.cc:491] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.211951 1908 raft_consensus.cc:513] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.212597 1908 leader_election.cc:290] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.215162 32052 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3] alter complete (version 1)
W20250623 08:04:41.217231 32052 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:39129 (request call id 176) took 2318 ms (client timeout 1000 ms). Trace:
W20250623 08:04:41.217336 32052 rpcz_store.cc:269] 0623 08:04:38.898540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.963918 (+ 65378us) service_pool.cc:225] Handling call
0623 08:04:39.000599 (+ 36681us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f3b58 00007f19c7640176 00007f19c763fea5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:41.217198 (+2216599us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:39.495307 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:39.495427 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:39.495433 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:39.495436 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:39.495560 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:39.495572 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:39.495577 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:39.495709 (+ 132us) tablet.cc:699] Row locks acquired
0623 08:04:39.495712 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:39.495746 (+ 34us) write_op.cc:270] Start()
0623 08:04:39.495776 (+ 30us) write_op.cc:276] Timestamp: P: 1750665879495714 usec, L: 0
0623 08:04:39.495780 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:39.495968 (+ 188us) log.cc:844] Serialized 901 byte log entry
0623 08:04:39.533592 (+ 37624us) op_driver.cc:464] REPLICATION: finished
0623 08:04:39.534035 (+ 443us) write_op.cc:301] APPLY: starting
0623 08:04:39.534050 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:39.534095 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:39.534137 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:39.534204 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:39.534238 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:39.534246 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:39.534321 (+ 75us) log.cc:844] Serialized 29 byte log entry
0623 08:04:39.534381 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.534401 (+ 20us) write_op.cc:454] Released schema lock
0623 08:04:39.534408 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:39.534475 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.534480 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:39.771467 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:39.771551 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:39.771557 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:39.771560 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:39.771648 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:39.771658 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:39.771662 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:39.771715 (+ 53us) tablet.cc:699] Row locks acquired
0623 08:04:39.771718 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:39.771750 (+ 32us) write_op.cc:270] Start()
0623 08:04:39.771779 (+ 29us) write_op.cc:276] Timestamp: P: 1750665879771689 usec, L: 0
0623 08:04:39.771782 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:39.771938 (+ 156us) log.cc:844] Serialized 356 byte log entry
0623 08:04:39.855263 (+ 83325us) op_driver.cc:464] REPLICATION: finished
0623 08:04:39.856254 (+ 991us) write_op.cc:301] APPLY: starting
0623 08:04:39.856265 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:39.856293 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:39.856326 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:39.856372 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:39.856421 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:39.856432 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:39.856495 (+ 63us) log.cc:844] Serialized 23 byte log entry
0623 08:04:39.856544 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.856559 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:39.856566 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:39.856635 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.856639 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":32,"spinlock_wait_cycles":81619200,"child_traces":[["op",{"apply.queue_time_us":397,"apply.run_cpu_time_us":432,"apply.run_wall_time_us":432,"num_ops":2,"prepare.queue_time_us":275314,"prepare.run_cpu_time_us":740,"prepare.run_wall_time_us":1074,"replication_time_us":37784}],["op",{"apply.queue_time_us":951,"apply.run_cpu_time_us":371,"apply.run_wall_time_us":370,"num_ops":1,"prepare.queue_time_us":214169,"prepare.run_cpu_time_us":602,"prepare.run_wall_time_us":10759,"replication_time_us":83452}]]}
I20250623 08:04:41.218683 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_bd4ee5f643fe4d9bab6b2f11ef0869cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:41.219388 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd4ee5f643fe4d9bab6b2f11ef0869cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.219530 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 52d0a356c74f4daea9093c998b4bf73e with delete_type TABLET_DATA_DELETED (replaced by tablet bb06a340f82047ac8ec3516edb08f4b6) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.219798 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 3249605158634f4eab5b9d61357c383f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.219971 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 724d662bcf754c74a8f3dff91ee258ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:38 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.220141 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 283bebc7063d437ba3e541d7dee3c0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.220264 32634 tablet_service.cc:1515] Processing DeleteTablet for tablet 36c8714cfbd84dd38cf5c0f305ef6843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:34 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.220317 32057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584] alter complete (version 2)
I20250623 08:04:41.220408 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.220608 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 74c1cdc83ffd4a09a163479ac58fd405
I20250623 08:04:41.220593 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 136e482995a24c5daec55a701f3091fa with delete_type TABLET_DATA_DELETED (replaced by tablet 3bfad68d1f20486997ea1b0a8d97206b) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.220798 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a96ae9190640dea11bd11a9f0e4372 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b1a601763e44934a7db2f53d05e43a5) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.220943 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet 6926c49b787941d98c647543bd3b9ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.221042 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet ecb422550efd40cf82bc92fca5c25eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.221040 32058 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 379) took 1711 ms. Trace:
I20250623 08:04:41.221141 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 025365d2fa0a43dd983d395a05cc2af1 with delete_type TABLET_DATA_DELETED (replaced by tablet a375f8ea383b44a9acd04fc2171d34ae) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.221138 32058 rpcz_store.cc:276] 0623 08:04:39.509559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.743755 (+234196us) service_pool.cc:225] Handling call
0623 08:04:39.768453 (+ 24698us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fa306 00007f19c7644956 00007f19c7644685 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:39.768766 (+ 313us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:39.768881 (+ 115us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:39.768933 (+ 52us) catalog_manager.cc:2799] Locking tablets
0623 08:04:39.924828 (+155895us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:39.944646 (+ 19818us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:39.956537 (+ 11891us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:41.220534 (+1263997us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:41.221009 (+ 475us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:39.928524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:39.928623 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:39.928628 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:39.928631 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:39.928722 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:39.928732 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:39.928737 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:39.928815 (+ 78us) tablet.cc:699] Row locks acquired
0623 08:04:39.928818 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:39.928845 (+ 27us) write_op.cc:270] Start()
0623 08:04:39.928872 (+ 27us) write_op.cc:276] Timestamp: P: 1750665879928825 usec, L: 0
0623 08:04:39.928875 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:39.929009 (+ 134us) log.cc:844] Serialized 725 byte log entry
0623 08:04:39.943132 (+ 14123us) op_driver.cc:464] REPLICATION: finished
0623 08:04:39.943476 (+ 344us) write_op.cc:301] APPLY: starting
0623 08:04:39.943497 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:39.943549 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:39.943595 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:39.943674 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:39.943719 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:39.943730 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:39.943790 (+ 60us) log.cc:844] Serialized 29 byte log entry
0623 08:04:39.943838 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.943854 (+ 16us) write_op.cc:454] Released schema lock
0623 08:04:39.943859 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:39.943928 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.943931 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":56253696,"child_traces":[["op",{"apply.queue_time_us":280,"apply.run_cpu_time_us":451,"apply.run_wall_time_us":452,"num_ops":2,"prepare.queue_time_us":3047,"prepare.run_cpu_time_us":546,"prepare.run_wall_time_us":795,"replication_time_us":14208}]]}
I20250623 08:04:41.221927 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 74c1cdc83ffd4a09a163479ac58fd405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.221900 32058 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0ac52a7e208745e980f3a49c46b3cdb8" } modify_external_catalogs: true
I20250623 08:04:41.224917 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 74c1cdc83ffd4a09a163479ac58fd405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.225790 1823 consensus_queue.cc:1035] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:41.226960 32054 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5af1dea3c6ca4f498ce6c039d72a14b7 consensus state has no leader
I20250623 08:04:41.228520 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet aa398354a9e44633857a75c0bbba08b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.228864 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 74c1cdc83ffd4a09a163479ac58fd405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:35542
W20250623 08:04:41.229168 32314 leader_election.cc:343] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 162308da898d4dde8ab5b1ab01d0f393
W20250623 08:04:41.229307 32056 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8a5e35c1974456b8079ab8907c2652e consensus state has no leader
W20250623 08:04:41.229995 32314 leader_election.cc:343] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 162308da898d4dde8ab5b1ab01d0f393
I20250623 08:04:41.230087 32314 leader_election.cc:304] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:41.231058 32057 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_161ad5ddd1cb40e0989e5c237a205db1 [id=7cc0919082d04fc4a02aa856a42e5529] alter complete (version 1)
I20250623 08:04:41.232265 1908 raft_consensus.cc:2747] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.235071 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 501b5233522f43c7aed3c9770adb5c03 with delete_type TABLET_DATA_DELETED (replaced by tablet 74c1cdc83ffd4a09a163479ac58fd405) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.236171 32618 tablet_service.cc:1515] Processing DeleteTablet for tablet f9f4829e103b460eb582beff3b966bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.236331 32622 tablet_service.cc:1515] Processing DeleteTablet for tablet b84b8b8713774ab4896842feb9fb075e with delete_type TABLET_DATA_DELETED (replaced by tablet a3576624314d4e85b411aab9109683f7) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:41.236411 32057 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:39129 (request call id 177) took 1086 ms (client timeout 1000 ms). Trace:
I20250623 08:04:41.236493 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 789d1c20aa72480ab71123c799bbde4c with delete_type TABLET_DATA_DELETED (replaced by tablet 0cca63991d5b4ef5bd8a9181d127ebe3) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.236631 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet ba3d2fa01d574dd992dc904ac024ba3a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.237172 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 92005abc61dd40ef9da8124cadc29b6a) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:41.236531 32057 rpcz_store.cc:269] 0623 08:04:40.149617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.617057 (+467440us) service_pool.cc:225] Handling call
0623 08:04:41.236375 (+619318us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:40.752941 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:40.753053 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:40.753057 (+ 4us) write_op.cc:435] Acquired schema lock
0623 08:04:40.753060 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:40.753169 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:40.753182 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:40.753186 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:40.753297 (+ 111us) tablet.cc:699] Row locks acquired
0623 08:04:40.753302 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:40.753335 (+ 33us) write_op.cc:270] Start()
0623 08:04:40.753364 (+ 29us) write_op.cc:276] Timestamp: P: 1750665880753277 usec, L: 0
0623 08:04:40.753370 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:40.753546 (+ 176us) log.cc:844] Serialized 901 byte log entry
0623 08:04:40.765129 (+ 11583us) op_driver.cc:464] REPLICATION: finished
0623 08:04:40.765263 (+ 134us) write_op.cc:301] APPLY: starting
0623 08:04:40.765282 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:40.765325 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:40.765357 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:40.765460 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:40.765511 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:40.765522 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:40.765618 (+ 96us) log.cc:844] Serialized 29 byte log entry
0623 08:04:40.765897 (+ 279us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.765924 (+ 27us) write_op.cc:454] Released schema lock
0623 08:04:40.765934 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:40.766023 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.766029 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:40.791141 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:40.791259 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:40.791264 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:40.791267 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:40.791371 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:40.791384 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:40.791389 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:40.791449 (+ 60us) tablet.cc:699] Row locks acquired
0623 08:04:40.791451 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:40.791482 (+ 31us) write_op.cc:270] Start()
0623 08:04:40.791511 (+ 29us) write_op.cc:276] Timestamp: P: 1750665880791422 usec, L: 0
0623 08:04:40.791516 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:40.791732 (+ 216us) log.cc:844] Serialized 356 byte log entry
0623 08:04:40.823566 (+ 31834us) op_driver.cc:464] REPLICATION: finished
0623 08:04:40.823915 (+ 349us) write_op.cc:301] APPLY: starting
0623 08:04:40.823937 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:40.823989 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:40.824021 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:40.824090 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:40.824139 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:40.824150 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:40.824240 (+ 90us) log.cc:844] Serialized 23 byte log entry
0623 08:04:40.824302 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.824324 (+ 22us) write_op.cc:454] Released schema lock
0623 08:04:40.824334 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:40.824430 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.824437 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:41.224621 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.224722 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.224729 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:41.224731 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:41.224840 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.224857 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.224861 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:41.224928 (+ 67us) tablet.cc:699] Row locks acquired
0623 08:04:41.224930 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:41.224965 (+ 35us) write_op.cc:270] Start()
0623 08:04:41.224997 (+ 32us) write_op.cc:276] Timestamp: P: 1750665881224917 usec, L: 0
0623 08:04:41.225000 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.225172 (+ 172us) log.cc:844] Serialized 356 byte log entry
0623 08:04:41.228264 (+ 3092us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.228383 (+ 119us) write_op.cc:301] APPLY: starting
0623 08:04:41.228396 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.228430 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.228457 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.228505 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.228538 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:41.228549 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:41.228616 (+ 67us) log.cc:844] Serialized 23 byte log entry
0623 08:04:41.228689 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.228708 (+ 19us) write_op.cc:454] Released schema lock
0623 08:04:41.228714 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:41.228788 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.228793 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2278,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":543,"apply.run_wall_time_us":743,"num_ops":2,"prepare.queue_time_us":15946,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":703,"raft.queue_time_us":33,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":409,"replication_time_us":11725}],["op",{"apply.queue_time_us":283,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":495,"num_ops":1,"prepare.queue_time_us":6010,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":679,"replication_time_us":32007}],["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":394,"apply.run_wall_time_us":394,"mutex_wait_us":3718,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":723,"prepare.run_wall_time_us":1087,"raft.queue_time_us":564,"raft.run_cpu_time_us":584,"raft.run_wall_time_us":583,"replication_time_us":3233,"spinlock_wait_cycles":20352}]]}
I20250623 08:04:41.238734 32057 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_88d6a5158a8d4ae192cf71e8eaffff54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:41.239560 32057 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88d6a5158a8d4ae192cf71e8eaffff54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.241971 1854 raft_consensus.cc:491] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.242148 1854 raft_consensus.cc:513] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:41.242799 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 44415.2us
I20250623 08:04:41.242861 1854 leader_election.cc:290] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.243410 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45293db3c40b4e4194445f12e7475e66" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.249156 339 raft_consensus.cc:357] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.249645 339 raft_consensus.cc:383] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.249666 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1be8f8557b9a4ae2917c37b8d01cedf6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.249848 339 raft_consensus.cc:738] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.252455 339 consensus_queue.cc:260] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.253583 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1be8f8557b9a4ae2917c37b8d01cedf6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:41.253729 32453 consensus_peers.cc:487] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:41.253517 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45293db3c40b4e4194445f12e7475e66" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.254127 32519 raft_consensus.cc:2466] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:41.259982 339 ts_tablet_manager.cc:1428] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.040s user 0.004s sys 0.000s
I20250623 08:04:41.261895 339 tablet_bootstrap.cc:492] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.266711 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd7f3d2e67e84bfcbd45a21658f69e7b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.267949 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd7f3d2e67e84bfcbd45a21658f69e7b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.270660 342 tablet_bootstrap.cc:492] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.270840 342 ts_tablet_manager.cc:1397] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.000s
I20250623 08:04:41.271590 342 raft_consensus.cc:357] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.271771 342 raft_consensus.cc:383] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.271832 342 raft_consensus.cc:738] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.272040 342 consensus_queue.cc:260] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.272536 342 ts_tablet_manager.cc:1428] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:41.272835 342 tablet_bootstrap.cc:492] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.274283 342 tablet_bootstrap.cc:654] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.295084 981 ts_tablet_manager.cc:1918] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.297576 981 log.cc:1199] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/48da1e4759bb4fea87a45ad8a428b6e7
I20250623 08:04:41.298228 981 ts_tablet_manager.cc:1939] T 48da1e4759bb4fea87a45ad8a428b6e7 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:41.298305 32056 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 380) took 1788 ms. Trace:
I20250623 08:04:41.298501 32056 rpcz_store.cc:276] 0623 08:04:39.509676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.744922 (+235246us) service_pool.cc:225] Handling call
0623 08:04:39.765769 (+ 20847us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fb5a6 00007f19c7646616 00007f19c7646345 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:39.766017 (+ 248us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:39.766238 (+ 221us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:39.766613 (+ 375us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:39.766692 (+ 79us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:39.861052 (+ 94360us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:41.298240 (+1437188us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:39.784565 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:39.784667 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:39.784672 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:39.784675 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:39.784758 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:39.784767 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:39.784771 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:39.784822 (+ 51us) tablet.cc:699] Row locks acquired
0623 08:04:39.784824 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:39.784858 (+ 34us) write_op.cc:270] Start()
0623 08:04:39.784889 (+ 31us) write_op.cc:276] Timestamp: P: 1750665879784795 usec, L: 0
0623 08:04:39.784893 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:39.785065 (+ 172us) log.cc:844] Serialized 405 byte log entry
0623 08:04:39.855401 (+ 70336us) op_driver.cc:464] REPLICATION: finished
0623 08:04:39.857048 (+ 1647us) write_op.cc:301] APPLY: starting
0623 08:04:39.857056 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:39.857079 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:39.857098 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:39.857121 (+ 23us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:39.857141 (+ 20us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:39.857148 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:39.857196 (+ 48us) log.cc:844] Serialized 23 byte log entry
0623 08:04:39.857225 (+ 29us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.857235 (+ 10us) write_op.cc:454] Released schema lock
0623 08:04:39.857239 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:39.857286 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:39.857290 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":47349504,"child_traces":[["op",{"apply.queue_time_us":1618,"apply.run_cpu_time_us":232,"apply.run_wall_time_us":233,"num_ops":1,"prepare.queue_time_us":17448,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":1941,"replication_time_us":70472}]]}
I20250623 08:04:41.299649 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_070eb9d8f3af4ed9bb823cf579ade9c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:41.300236 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_070eb9d8f3af4ed9bb823cf579ade9c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.301792 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 243) took 4718 ms. Trace:
I20250623 08:04:41.302877 32058 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0ac52a7e208745e980f3a49c46b3cdb8" } modify_external_catalogs: true
I20250623 08:04:41.301877 981 rpcz_store.cc:276] 0623 08:04:36.583036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.583098 (+ 62us) service_pool.cc:225] Handling call
0623 08:04:41.150086 (+4566988us) ts_tablet_manager.cc:1145] Deleting tablet 48da1e4759bb4fea87a45ad8a428b6e7
0623 08:04:41.150094 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.233259 (+ 83165us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.294955 (+ 61696us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.301770 (+ 6815us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4566759}
I20250623 08:04:41.303709 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 48da1e4759bb4fea87a45ad8a428b6e7 (table default.Table_c382db2a0c144ec4abd51b3e927e0a17 [id=461d38c05dc24245b0c59cb73b42a394]) successfully deleted
I20250623 08:04:41.307126 32054 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 393) took 1368 ms. Trace:
I20250623 08:04:41.307251 32054 rpcz_store.cc:276] 0623 08:04:39.938811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.826702 (+887891us) service_pool.cc:225] Handling call
0623 08:04:40.992707 (+166005us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fb5a6 00007f19c7646616 00007f19c7646345 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:40.993036 (+ 329us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:40.993253 (+ 217us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:40.993483 (+ 230us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:40.993577 (+ 94us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:41.006103 (+ 12526us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:41.307056 (+300953us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:40.995718 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:40.995840 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:40.995846 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:40.995848 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:40.995956 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:40.995971 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:40.995979 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:40.996050 (+ 71us) tablet.cc:699] Row locks acquired
0623 08:04:40.996054 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:40.996088 (+ 34us) write_op.cc:270] Start()
0623 08:04:40.996122 (+ 34us) write_op.cc:276] Timestamp: P: 1750665880996063 usec, L: 0
0623 08:04:40.996126 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:40.996290 (+ 164us) log.cc:844] Serialized 405 byte log entry
0623 08:04:41.003732 (+ 7442us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.005045 (+ 1313us) write_op.cc:301] APPLY: starting
0623 08:04:41.005063 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.005105 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.005165 (+ 60us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.005222 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.005256 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:41.005264 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:41.005355 (+ 91us) log.cc:844] Serialized 23 byte log entry
0623 08:04:41.005451 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.005472 (+ 21us) write_op.cc:454] Released schema lock
0623 08:04:41.005479 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:41.005546 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.005550 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":381076352,"child_traces":[["op",{"apply.queue_time_us":1233,"apply.run_cpu_time_us":500,"apply.run_wall_time_us":501,"num_ops":1,"prepare.queue_time_us":1634,"prepare.run_cpu_time_us":693,"prepare.run_wall_time_us":693,"raft.queue_time_us":398,"raft.run_cpu_time_us":492,"raft.run_wall_time_us":492,"replication_time_us":7576}]]}
I20250623 08:04:41.308074 32052 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 394) took 1369 ms. Trace:
I20250623 08:04:41.308241 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 244) took 4725 ms. Trace:
I20250623 08:04:41.308307 981 rpcz_store.cc:276] 0623 08:04:36.583158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.583367 (+ 209us) service_pool.cc:225] Handling call
0623 08:04:41.308088 (+4724721us) ts_tablet_manager.cc:1145] Deleting tablet b61b02cf2904404f8c0ffef7d268fefc
0623 08:04:41.308101 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.308224 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4724505}
I20250623 08:04:41.308207 32052 rpcz_store.cc:276] 0623 08:04:39.938952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.218453 (+1279501us) service_pool.cc:225] Handling call
0623 08:04:41.296023 (+ 77570us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:41.296647 (+ 624us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:41.307816 (+ 11169us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:41.307927 (+ 111us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:41.308011 (+ 84us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.297246 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.297385 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.297391 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:41.297393 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:41.297535 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.297554 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.297562 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:41.297670 (+ 108us) tablet.cc:699] Row locks acquired
0623 08:04:41.297674 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:41.297718 (+ 44us) write_op.cc:270] Start()
0623 08:04:41.297762 (+ 44us) write_op.cc:276] Timestamp: P: 1750665881297656 usec, L: 0
0623 08:04:41.297768 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.298008 (+ 240us) log.cc:844] Serialized 433 byte log entry
0623 08:04:41.301978 (+ 3970us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.302079 (+ 101us) write_op.cc:301] APPLY: starting
0623 08:04:41.302095 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.302137 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.302179 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.302287 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.302323 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:41.302331 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:41.302426 (+ 95us) log.cc:844] Serialized 29 byte log entry
0623 08:04:41.302488 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.302505 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:41.302511 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:41.302577 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.302581 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":489,"apply.run_wall_time_us":489,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":895,"prepare.run_wall_time_us":895,"raft.queue_time_us":2647,"raft.run_cpu_time_us":488,"raft.run_wall_time_us":488,"replication_time_us":4159}]]}
W20250623 08:04:41.308750 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet b61b02cf2904404f8c0ffef7d268fefc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b61b02cf2904404f8c0ffef7d268fefc already in progress: creating tablet
I20250623 08:04:41.308903 32052 master_service.cc:948] ReplaceTablet: received request to replace tablet f0e24d8d890d4d1db051f172b3ec96ec from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:41.309110 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 245) took 4725 ms. Trace:
I20250623 08:04:41.309177 981 rpcz_store.cc:276] 0623 08:04:36.583274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.583359 (+ 85us) service_pool.cc:225] Handling call
0623 08:04:41.309014 (+4725655us) ts_tablet_manager.cc:1145] Deleting tablet b720fdea539f423ba2d2aa9f89b09ae9
0623 08:04:41.309022 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.309095 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4725404}
I20250623 08:04:41.309288 32050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 3136ec837bea4619ae3995ceec9b24d6
W20250623 08:04:41.309561 32043 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet b720fdea539f423ba2d2aa9f89b09ae9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b720fdea539f423ba2d2aa9f89b09ae9 already in progress: creating tablet
I20250623 08:04:41.309828 32050 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 392) took 1374 ms. Trace:
I20250623 08:04:41.310213 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 3136ec837bea4619ae3995ceec9b24d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:40 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.310297 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet 3136ec837bea4619ae3995ceec9b24d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:40 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.309931 32050 rpcz_store.cc:276] 0623 08:04:39.935464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.826262 (+890798us) service_pool.cc:225] Handling call
0623 08:04:40.992807 (+166545us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fa306 00007f19c7644956 00007f19c7644685 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:40.993160 (+ 353us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:40.993292 (+ 132us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:40.993346 (+ 54us) catalog_manager.cc:2799] Locking tablets
0623 08:04:41.260356 (+267010us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:41.265797 (+ 5441us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:41.309193 (+ 43396us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:41.309246 (+ 53us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:41.309764 (+ 518us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.261146 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.261254 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.261259 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:41.261263 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:41.261370 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.261381 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.261386 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:41.261487 (+ 101us) tablet.cc:699] Row locks acquired
0623 08:04:41.261490 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:41.261523 (+ 33us) write_op.cc:270] Start()
0623 08:04:41.261554 (+ 31us) write_op.cc:276] Timestamp: P: 1750665881261469 usec, L: 0
0623 08:04:41.261558 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.261715 (+ 157us) log.cc:844] Serialized 725 byte log entry
0623 08:04:41.264965 (+ 3250us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.265073 (+ 108us) write_op.cc:301] APPLY: starting
0623 08:04:41.265087 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.265129 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.265157 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.265223 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.265262 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:41.265269 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:41.265345 (+ 76us) log.cc:844] Serialized 29 byte log entry
0623 08:04:41.265440 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.265464 (+ 24us) write_op.cc:454] Released schema lock
0623 08:04:41.265470 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:41.265538 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.265542 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":382324992,"child_traces":[["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":455,"apply.run_wall_time_us":454,"num_ops":2,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":683,"raft.queue_time_us":1184,"raft.run_cpu_time_us":623,"raft.run_wall_time_us":623,"replication_time_us":3377}]]}
I20250623 08:04:41.310626 32050 master_service.cc:948] ReplaceTablet: received request to replace tablet 4c9d1c8a67e149478a0d69842a4eb5b2 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:41.310854 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 3136ec837bea4619ae3995ceec9b24d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:40 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.311340 981 tablet_replica.cc:331] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:41.311542 981 raft_consensus.cc:2241] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:41.311702 981 raft_consensus.cc:2270] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.320575 981 ts_tablet_manager.cc:1905] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.323351 32056 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 397) took 1376 ms. Trace:
I20250623 08:04:41.323488 32056 rpcz_store.cc:276] 0623 08:04:39.946881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.299425 (+1352544us) service_pool.cc:225] Handling call
0623 08:04:41.300329 (+ 904us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:41.300798 (+ 469us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:41.323141 (+ 22343us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:41.323234 (+ 93us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:41.323312 (+ 78us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.302192 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.302283 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.302289 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:41.302291 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:41.302367 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.302377 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.302408 (+ 31us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:41.302482 (+ 74us) tablet.cc:699] Row locks acquired
0623 08:04:41.302485 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:41.302515 (+ 30us) write_op.cc:270] Start()
0623 08:04:41.302543 (+ 28us) write_op.cc:276] Timestamp: P: 1750665881302460 usec, L: 0
0623 08:04:41.302548 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.302728 (+ 180us) log.cc:844] Serialized 433 byte log entry
0623 08:04:41.320965 (+ 18237us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.321411 (+ 446us) write_op.cc:301] APPLY: starting
0623 08:04:41.321427 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.321477 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.321530 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.321629 (+ 99us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.321661 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:41.321668 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:41.321759 (+ 91us) log.cc:844] Serialized 29 byte log entry
0623 08:04:41.321814 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.321830 (+ 16us) write_op.cc:454] Released schema lock
0623 08:04:41.321835 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:41.321895 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.321899 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":397,"apply.run_cpu_time_us":481,"apply.run_wall_time_us":481,"num_ops":2,"prepare.queue_time_us":967,"prepare.run_cpu_time_us":724,"prepare.run_wall_time_us":18789,"raft.queue_time_us":588,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"replication_time_us":18380}]]}
I20250623 08:04:41.324115 32056 master_service.cc:948] ReplaceTablet: received request to replace tablet 724d662bcf754c74a8f3dff91ee258ae from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:41.330814 32589 leader_election.cc:304] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.331307 32589 leader_election.cc:304] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:41.331610 32589 leader_election.cc:343] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.331732 32589 leader_election.cc:304] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:41.332188 32589 leader_election.cc:343] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.332345 32589 leader_election.cc:304] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:41.332249 1863 consensus_queue.cc:1035] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:41.332571 32589 leader_election.cc:343] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.332878 1863 raft_consensus.cc:2802] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:41.332981 1863 raft_consensus.cc:491] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:41.333037 1863 raft_consensus.cc:3058] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:41.333164 32589 leader_election.cc:343] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:41.333463 32589 leader_election.cc:343] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.333554 32589 leader_election.cc:304] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:41.333921 32589 leader_election.cc:343] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.334152 1877 raft_consensus.cc:2802] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.334226 1877 raft_consensus.cc:695] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:41.334475 1854 raft_consensus.cc:2747] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.334609 1854 consensus_queue.cc:1035] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.335059 1854 raft_consensus.cc:2802] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:41.335120 1854 raft_consensus.cc:491] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:41.335196 1854 raft_consensus.cc:3058] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.335659 1903 raft_consensus.cc:2747] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.334007 32589 leader_election.cc:304] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.339591 1902 raft_consensus.cc:2747] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.340818 1877 consensus_queue.cc:237] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.346714 1908 raft_consensus.cc:491] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.346885 1908 raft_consensus.cc:513] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.347570 1908 leader_election.cc:290] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:41.348623 32314 leader_election.cc:343] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 8dc696236a15420e9666cca708068fb2
W20250623 08:04:41.350359 32314 leader_election.cc:343] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 8dc696236a15420e9666cca708068fb2
I20250623 08:04:41.350497 32314 leader_election.cc:304] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:41.350726 1908 raft_consensus.cc:2747] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250623 08:04:41.356487 343 tablet_bootstrap.cc:492] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:41.356834 343 ts_tablet_manager.cc:1397] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.166s user 0.005s sys 0.002s
I20250623 08:04:41.359673 1854 raft_consensus.cc:513] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.360605 1771 raft_consensus.cc:491] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.360790 1771 raft_consensus.cc:513] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.361478 1771 leader_election.cc:290] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.362169 1854 leader_election.cc:290] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.366026 979 ts_tablet_manager.cc:1918] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.366178 979 log.cc:1199] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/e9f8aa424bbb4aa8a907259270d46656
I20250623 08:04:41.366621 979 ts_tablet_manager.cc:1939] T e9f8aa424bbb4aa8a907259270d46656 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:41.367568 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 331) took 2141 ms. Trace:
I20250623 08:04:41.368427 979 rpcz_store.cc:276] 0623 08:04:39.225634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.238516 (+ 12882us) service_pool.cc:225] Handling call
0623 08:04:41.126720 (+1888204us) ts_tablet_manager.cc:1145] Deleting tablet e9f8aa424bbb4aa8a907259270d46656
0623 08:04:41.126728 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.329985 (+203257us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.365922 (+ 35937us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.367551 (+ 1629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1888021}
I20250623 08:04:41.369134 1863 raft_consensus.cc:513] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.369349 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet e9f8aa424bbb4aa8a907259270d46656 (table default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]) successfully deleted
I20250623 08:04:41.369910 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 337) took 2058 ms. Trace:
I20250623 08:04:41.380288 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c14e325fa5f4f848748c346bc25dde0" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.381327 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c14e325fa5f4f848748c346bc25dde0" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.382265 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45293db3c40b4e4194445f12e7475e66" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:41.383141 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45293db3c40b4e4194445f12e7475e66" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:41.383291 32514 raft_consensus.cc:3058] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.386451 32514 raft_consensus.cc:2466] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:41.389175 979 rpcz_store.cc:276] 0623 08:04:39.311274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.313489 (+ 2215us) service_pool.cc:225] Handling call
0623 08:04:41.369791 (+2056302us) ts_tablet_manager.cc:1145] Deleting tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159
0623 08:04:41.369802 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.369896 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2056012}
W20250623 08:04:41.389792 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 already in progress: creating tablet
I20250623 08:04:41.390314 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 336) took 2079 ms. Trace:
I20250623 08:04:41.390436 979 rpcz_store.cc:276] 0623 08:04:39.311127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.324875 (+ 13748us) service_pool.cc:225] Handling call
0623 08:04:41.390180 (+2065305us) ts_tablet_manager.cc:1145] Deleting tablet 1b213c8c685c43868df1e814c79fc61f
0623 08:04:41.390191 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.390295 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2065021}
W20250623 08:04:41.390926 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 1b213c8c685c43868df1e814c79fc61f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b213c8c685c43868df1e814c79fc61f already in progress: creating tablet
I20250623 08:04:41.391330 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 338) took 2015 ms. Trace:
I20250623 08:04:41.391556 979 rpcz_store.cc:276] 0623 08:04:39.375549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.377051 (+ 1502us) service_pool.cc:225] Handling call
0623 08:04:41.391233 (+2014182us) ts_tablet_manager.cc:1145] Deleting tablet 803d3607d75443568b7ddc9f0c0bdc90
0623 08:04:41.391242 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.391315 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2014001}
W20250623 08:04:41.392006 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 803d3607d75443568b7ddc9f0c0bdc90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 803d3607d75443568b7ddc9f0c0bdc90 already in progress: creating tablet
I20250623 08:04:41.392359 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 340) took 1942 ms. Trace:
I20250623 08:04:41.392479 979 rpcz_store.cc:276] 0623 08:04:39.449733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.449791 (+ 58us) service_pool.cc:225] Handling call
0623 08:04:41.392266 (+1942475us) ts_tablet_manager.cc:1145] Deleting tablet 90ceb0f8d5c94c4e9427042d8a35e16c
0623 08:04:41.392275 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.392343 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942300}
W20250623 08:04:41.393348 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 90ceb0f8d5c94c4e9427042d8a35e16c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90ceb0f8d5c94c4e9427042d8a35e16c already in progress: creating tablet
W20250623 08:04:41.393929 32314 consensus_peers.cc:487] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9f8aa424bbb4aa8a907259270d46656. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:41.392952 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 343) took 1598 ms. Trace:
I20250623 08:04:41.394268 979 rpcz_store.cc:276] 0623 08:04:39.794889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.795107 (+ 218us) service_pool.cc:225] Handling call
0623 08:04:41.392847 (+1597740us) ts_tablet_manager.cc:1145] Deleting tablet aface2a39d764a6cb6e4965867ab7b38
0623 08:04:41.392858 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.392940 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597475}
I20250623 08:04:41.395171 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5af1dea3c6ca4f498ce6c039d72a14b7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:41.396142 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5af1dea3c6ca4f498ce6c039d72a14b7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:41.396312 32516 raft_consensus.cc:3058] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.222697 952 ts_tablet_manager.cc:1918] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.398882 952 log.cc:1199] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/d6ed4cb252b1452087f8ee53ec78d985
I20250623 08:04:41.399407 32380 raft_consensus.cc:3058] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.399587 952 ts_tablet_manager.cc:1939] T d6ed4cb252b1452087f8ee53ec78d985 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:41.400609 339 tablet_bootstrap.cc:654] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.401705 1359 raft_consensus.cc:491] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.401882 1359 raft_consensus.cc:513] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.402662 1359 leader_election.cc:290] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.403242 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9d1c8a67e149478a0d69842a4eb5b2" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:41.403932 32453 leader_election.cc:343] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.405472 1359 consensus_queue.cc:1035] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.405877 32516 raft_consensus.cc:2466] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:41.414907 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 190) took 6110 ms. Trace:
I20250623 08:04:41.415185 952 rpcz_store.cc:276] 0623 08:04:35.304531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.313779 (+ 9248us) service_pool.cc:225] Handling call
0623 08:04:41.208728 (+5894949us) ts_tablet_manager.cc:1145] Deleting tablet d6ed4cb252b1452087f8ee53ec78d985
0623 08:04:41.208737 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.219633 (+ 10896us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.222600 (+ 2967us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.414880 (+192280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5894727}
I20250623 08:04:41.415819 952 tablet_replica.cc:331] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:41.415984 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet d6ed4cb252b1452087f8ee53ec78d985 (table default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]) successfully deleted
I20250623 08:04:41.416203 952 raft_consensus.cc:2241] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:41.416666 952 raft_consensus.cc:2270] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.377441 1789 consensus_queue.cc:1035] T 0861e45cf546487d9812caa06e2704eb P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.403316 343 raft_consensus.cc:357] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.403667 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9d1c8a67e149478a0d69842a4eb5b2" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:41.425913 343 raft_consensus.cc:383] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.426014 343 raft_consensus.cc:738] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.426040 32642 raft_consensus.cc:2466] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:41.426394 343 consensus_queue.cc:260] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:41.426620 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet aface2a39d764a6cb6e4965867ab7b38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aface2a39d764a6cb6e4965867ab7b38 already in progress: creating tablet
I20250623 08:04:41.427465 343 ts_tablet_manager.cc:1428] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.067s user 0.001s sys 0.001s
I20250623 08:04:41.370124 1863 leader_election.cc:290] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:41.414724 32589 leader_election.cc:343] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.427924 343 tablet_bootstrap.cc:492] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:41.427951 32589 leader_election.cc:304] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:41.428303 32589 leader_election.cc:304] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:41.428530 1789 raft_consensus.cc:2802] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.428651 1789 raft_consensus.cc:695] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:41.428963 1789 consensus_queue.cc:237] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.430034 1854 raft_consensus.cc:2802] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.430130 1854 raft_consensus.cc:695] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:41.430317 1854 consensus_queue.cc:237] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.432458 1789 consensus_queue.cc:1035] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.432864 1789 consensus_queue.cc:1035] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:41.433396 32589 leader_election.cc:343] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.433533 32589 leader_election.cc:304] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.433749 1789 raft_consensus.cc:2747] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:41.434906 32589 leader_election.cc:343] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:41.435341 32589 leader_election.cc:343] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.444671 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 344) took 1649 ms. Trace:
I20250623 08:04:41.444832 979 rpcz_store.cc:276] 0623 08:04:39.795052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.795405 (+ 353us) service_pool.cc:225] Handling call
0623 08:04:41.427620 (+1632215us) ts_tablet_manager.cc:1145] Deleting tablet 1aa604f500984e3b969e15592cdbed6f
0623 08:04:41.444531 (+ 16911us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.444646 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1632037}
W20250623 08:04:41.445394 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 1aa604f500984e3b969e15592cdbed6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1aa604f500984e3b969e15592cdbed6f already in progress: creating tablet
I20250623 08:04:41.446020 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 346) took 1541 ms. Trace:
I20250623 08:04:41.446637 979 rpcz_store.cc:276] 0623 08:04:39.904448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.904700 (+ 252us) service_pool.cc:225] Handling call
0623 08:04:41.445916 (+1541216us) ts_tablet_manager.cc:1145] Deleting tablet 185c2bf7b9254947bb76b7fe0924559e
0623 08:04:41.445927 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.446005 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1541000}
W20250623 08:04:41.447743 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 185c2bf7b9254947bb76b7fe0924559e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 185c2bf7b9254947bb76b7fe0924559e already in progress: creating tablet
I20250623 08:04:41.448365 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 347) took 1493 ms. Trace:
I20250623 08:04:41.449132 979 rpcz_store.cc:276] 0623 08:04:39.955147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.955206 (+ 59us) service_pool.cc:225] Handling call
0623 08:04:41.448232 (+1493026us) ts_tablet_manager.cc:1145] Deleting tablet 9dbbdbebdf3b4b50bda31a94fc0de36f
0623 08:04:41.448245 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.448345 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492831}
W20250623 08:04:41.450127 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9dbbdbebdf3b4b50bda31a94fc0de36f already in progress: creating tablet
I20250623 08:04:41.450614 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 348) took 1492 ms. Trace:
I20250623 08:04:41.451151 979 rpcz_store.cc:276] 0623 08:04:39.957778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.957832 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:41.450521 (+1492689us) ts_tablet_manager.cc:1145] Deleting tablet 30dfa986dfd44658a0ffb07796030520
0623 08:04:41.450530 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.450599 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492515}
W20250623 08:04:41.451993 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 30dfa986dfd44658a0ffb07796030520 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30dfa986dfd44658a0ffb07796030520 already in progress: creating tablet
W20250623 08:04:41.452746 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet a383d10d53b04d5699dd3c4bad8878c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a383d10d53b04d5699dd3c4bad8878c7 already in progress: creating tablet
W20250623 08:04:41.453817 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 already in progress: creating tablet
I20250623 08:04:41.463186 1784 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: acquiring CA information for follower catalog manager: success
I20250623 08:04:41.466558 32050 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36116 (request call id 400) took 1273 ms. Trace:
I20250623 08:04:41.466595 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_0e43b0bc6b7b47b1b59882856aaa9736"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:41.466701 32050 rpcz_store.cc:276] 0623 08:04:40.192577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.310545 (+1117968us) service_pool.cc:225] Handling call
0623 08:04:41.466278 (+155733us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d080605b 00007f19c764f5d7 00007f19c764f305 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:41.466499 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":357151616}
I20250623 08:04:41.467231 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_5afa01a4e2fa4b7c97e51a681da648d0" } new_table_name: "default.Table_d0e3d64801be4520a9c8d9c6b99f78ef" modify_external_catalogs: true
W20250623 08:04:41.467407 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e43b0bc6b7b47b1b59882856aaa9736 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.469566 32052 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36116 (request call id 399) took 1339 ms. Trace:
I20250623 08:04:41.469496 32056 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_969668212d4f422ca8f2f34266315161" } new_table_name: "default.Table_df8f082e8b3c45178294a9ab36df0fdc" modify_external_catalogs: true
I20250623 08:04:41.469658 32052 rpcz_store.cc:276] 0623 08:04:40.130137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.308820 (+1178683us) service_pool.cc:225] Handling call
0623 08:04:41.469321 (+160501us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d080605b 00007f19c764f5d7 00007f19c764f305 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:41.469501 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":368386432}
I20250623 08:04:41.470021 32052 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0b9c0d17ddcd422eb8035e69884f3da6" } modify_external_catalogs: true
I20250623 08:04:41.470173 32052 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_0b9c0d17ddcd422eb8035e69884f3da6" } modify_external_catalogs: true
I20250623 08:04:41.471771 339 tablet_bootstrap.cc:492] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.471957 339 ts_tablet_manager.cc:1397] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.211s user 0.005s sys 0.000s
I20250623 08:04:41.472656 339 raft_consensus.cc:357] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.472895 339 raft_consensus.cc:383] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.472980 339 raft_consensus.cc:738] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.473326 339 consensus_queue.cc:260] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.473627 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:41.474174 339 ts_tablet_manager.cc:1428] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:41.474442 339 tablet_bootstrap.cc:492] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.475808 339 tablet_bootstrap.cc:654] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.481770 1908 consensus_queue.cc:1035] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.427345 32453 leader_election.cc:304] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:41.484134 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 56824.9us
I20250623 08:04:41.484189 952 ts_tablet_manager.cc:1905] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.484985 1359 raft_consensus.cc:2802] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:41.485073 1359 raft_consensus.cc:491] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:41.485142 1359 raft_consensus.cc:3058] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.487778 1359 raft_consensus.cc:513] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.488332 1359 leader_election.cc:290] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.490689 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9d1c8a67e149478a0d69842a4eb5b2" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:41.491775 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9d1c8a67e149478a0d69842a4eb5b2" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:41.491952 32656 raft_consensus.cc:3058] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.494601 32656 raft_consensus.cc:2466] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:41.495090 32453 leader_election.cc:343] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.496583 1823 consensus_queue.cc:1035] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.499671 32453 leader_election.cc:304] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.500373 1823 raft_consensus.cc:2802] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.500522 1823 raft_consensus.cc:695] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:41.500844 1823 consensus_queue.cc:237] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.507669 32057 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 396) took 1561 ms. Trace:
I20250623 08:04:41.507841 32057 rpcz_store.cc:276] 0623 08:04:39.945873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.238538 (+1292665us) service_pool.cc:225] Handling call
0623 08:04:41.310908 (+ 72370us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:41.311501 (+ 593us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:41.507354 (+195853us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:41.507497 (+ 143us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:41.507610 (+ 113us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.465016 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.465125 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.465131 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:41.465134 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:41.465221 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.465232 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.465237 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:41.465322 (+ 85us) tablet.cc:699] Row locks acquired
0623 08:04:41.465324 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:41.465357 (+ 33us) write_op.cc:270] Start()
0623 08:04:41.474236 (+ 8879us) write_op.cc:276] Timestamp: P: 1750665881465324 usec, L: 0
0623 08:04:41.474241 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.474394 (+ 153us) log.cc:844] Serialized 433 byte log entry
0623 08:04:41.506544 (+ 32150us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.506785 (+ 241us) write_op.cc:301] APPLY: starting
0623 08:04:41.506802 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.506852 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.506883 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.507020 (+ 137us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.507049 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:41.507055 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:41.507126 (+ 71us) log.cc:844] Serialized 29 byte log entry
0623 08:04:41.507192 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.507210 (+ 18us) write_op.cc:454] Released schema lock
0623 08:04:41.507218 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":162,"apply.run_cpu_time_us":501,"apply.run_wall_time_us":526,"num_ops":2,"prepare.queue_time_us":153110,"prepare.run_cpu_time_us":663,"prepare.run_wall_time_us":14137,"raft.queue_time_us":846,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":216,"replication_time_us":32272}]]}
I20250623 08:04:41.510596 32057 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_7eaaf57b62b345609a21707908207acd" } new_table_name: "default.Table_1b5a331260574f8eb6a61a556aee3a8e" modify_external_catalogs: true
I20250623 08:04:41.516304 342 tablet_bootstrap.cc:492] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.516448 342 ts_tablet_manager.cc:1397] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.244s user 0.075s sys 0.106s
I20250623 08:04:41.517125 342 raft_consensus.cc:357] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.517334 342 raft_consensus.cc:383] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.517405 342 raft_consensus.cc:738] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.517621 342 consensus_queue.cc:260] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.521885 342 ts_tablet_manager.cc:1428] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:41.522183 342 tablet_bootstrap.cc:492] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.524003 342 tablet_bootstrap.cc:654] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.527169 1908 raft_consensus.cc:491] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.527328 1908 raft_consensus.cc:513] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.528009 1908 leader_election.cc:290] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.528652 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a96ae9190640dea11bd11a9f0e4372" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.528827 32514 raft_consensus.cc:2466] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:41.529353 32314 leader_election.cc:304] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:41.529708 1682 raft_consensus.cc:2802] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:41.529783 1682 raft_consensus.cc:491] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:41.529839 1682 raft_consensus.cc:3058] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.532853 1682 raft_consensus.cc:513] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.533432 1682 leader_election.cc:290] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.533980 952 ts_tablet_manager.cc:1918] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.535516 1682 consensus_queue.cc:1035] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:41.535686 32314 leader_election.cc:343] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: f4a96ae9190640dea11bd11a9f0e4372
W20250623 08:04:41.537102 32314 leader_election.cc:343] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: f4a96ae9190640dea11bd11a9f0e4372
I20250623 08:04:41.544075 339 tablet_bootstrap.cc:492] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.544118 1771 consensus_queue.cc:1035] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.544237 339 ts_tablet_manager.cc:1397] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.000s
I20250623 08:04:41.544955 339 raft_consensus.cc:357] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.546857 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4a96ae9190640dea11bd11a9f0e4372" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:41.548496 32519 raft_consensus.cc:3058] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.552033 979 tablet_replica.cc:331] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:41.552246 979 raft_consensus.cc:2241] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:41.545111 339 raft_consensus.cc:383] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.552368 339 raft_consensus.cc:738] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:41.552508 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d5cd316e6f4447be958611ab07ed31fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5cd316e6f4447be958611ab07ed31fc already in progress: creating tablet
I20250623 08:04:41.552628 339 consensus_queue.cc:260] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.553244 339 ts_tablet_manager.cc:1428] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250623 08:04:41.553576 339 tablet_bootstrap.cc:492] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.554010 32054 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36116 (request call id 398) took 1493 ms. Trace:
I20250623 08:04:41.554124 32054 rpcz_store.cc:276] 0623 08:04:40.060999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.308223 (+1247224us) service_pool.cc:225] Handling call
0623 08:04:41.466474 (+158251us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f9313 00007f19c7642c96 00007f19c76429c5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:41.467531 (+ 1057us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:41.468129 (+ 598us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:41.553745 (+ 85616us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:41.553854 (+ 109us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:41.553949 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.483584 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.483679 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.483684 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:41.483686 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:41.483770 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.483779 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.483785 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:41.483860 (+ 75us) tablet.cc:699] Row locks acquired
0623 08:04:41.483863 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:41.483893 (+ 30us) write_op.cc:270] Start()
0623 08:04:41.483935 (+ 42us) write_op.cc:276] Timestamp: P: 1750665881483857 usec, L: 0
0623 08:04:41.483939 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.484074 (+ 135us) log.cc:844] Serialized 433 byte log entry
0623 08:04:41.548823 (+ 64749us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.550233 (+ 1410us) write_op.cc:301] APPLY: starting
0623 08:04:41.550248 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.550285 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.550320 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.550437 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.550494 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:41.550505 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:41.550585 (+ 80us) log.cc:844] Serialized 29 byte log entry
0623 08:04:41.550651 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.550673 (+ 22us) write_op.cc:454] Released schema lock
0623 08:04:41.550682 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":363017984,"child_traces":[["op",{"apply.queue_time_us":1354,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":524,"num_ops":2,"prepare.queue_time_us":14851,"prepare.run_cpu_time_us":583,"prepare.run_wall_time_us":8166,"raft.queue_time_us":769,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":190,"replication_time_us":64859}]]}
I20250623 08:04:41.556190 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_9a49c84f75b84594ab81fcf31da3991f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:41.556785 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a49c84f75b84594ab81fcf31da3991f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.564230 339 tablet_bootstrap.cc:654] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.570572 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 48ea78fb01d646d681404009eab9b050 (DEFAULT_TABLE table=default.Table_6b3d1637c4c1443089f04128e190256f [id=3f48141767f441dbbdf5a7742a16176f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:41.571004 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ea78fb01d646d681404009eab9b050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:41.571632 1682 raft_consensus.cc:491] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.571753 1682 raft_consensus.cc:513] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.572266 1682 leader_election.cc:290] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:41.573660 32314 leader_election.cc:343] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 205da880685b49b3b8669f73dc2c563b
W20250623 08:04:41.573838 32314 leader_election.cc:343] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 205da880685b49b3b8669f73dc2c563b
I20250623 08:04:41.573918 32314 leader_election.cc:304] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:41.574250 1682 raft_consensus.cc:2747] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.575495 32358 tablet_service.cc:1468] Processing CreateTablet for tablet 48ea78fb01d646d681404009eab9b050 (DEFAULT_TABLE table=default.Table_6b3d1637c4c1443089f04128e190256f [id=3f48141767f441dbbdf5a7742a16176f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:41.575850 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ea78fb01d646d681404009eab9b050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:41.585021 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_bc37bf6911b5401c904eef106dde3592"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:41.585839 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc37bf6911b5401c904eef106dde3592 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.587209 32500 tablet_service.cc:1468] Processing CreateTablet for tablet c098dcb1c55c41ebbf6229cfe8d24a44 (DEFAULT_TABLE table=default.Table_5cfbfaf89e6a4bdf94cef7019cf9c5e4 [id=5c35ac21e1964919acdfdc5e72e5b6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:41.587605 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c098dcb1c55c41ebbf6229cfe8d24a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:41.594113 32358 tablet_service.cc:1468] Processing CreateTablet for tablet c098dcb1c55c41ebbf6229cfe8d24a44 (DEFAULT_TABLE table=default.Table_5cfbfaf89e6a4bdf94cef7019cf9c5e4 [id=5c35ac21e1964919acdfdc5e72e5b6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:41.594560 32358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c098dcb1c55c41ebbf6229cfe8d24a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:41.596968 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_43ff9ceffb4042b5ab1b041b76bdfc96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:41.597707 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43ff9ceffb4042b5ab1b041b76bdfc96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.602727 342 tablet_bootstrap.cc:492] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.602953 342 ts_tablet_manager.cc:1397] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.012s
I20250623 08:04:41.603854 342 raft_consensus.cc:357] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.604053 342 raft_consensus.cc:383] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.604143 342 raft_consensus.cc:738] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.604369 342 consensus_queue.cc:260] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.605048 342 ts_tablet_manager.cc:1428] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:41.605418 342 tablet_bootstrap.cc:492] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.534089 952 log.cc:1199] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f4a0b9402a264800a1ea39707c10da0c
I20250623 08:04:41.607626 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_cddb0cb673ab4aacb6e69f181369ca8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:41.608489 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cddb0cb673ab4aacb6e69f181369ca8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.609176 952 ts_tablet_manager.cc:1939] T f4a0b9402a264800a1ea39707c10da0c P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:41.610198 342 tablet_bootstrap.cc:654] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.615684 339 tablet_bootstrap.cc:492] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.616067 339 ts_tablet_manager.cc:1397] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.001s
I20250623 08:04:41.571187 32627 tablet_service.cc:1468] Processing CreateTablet for tablet c098dcb1c55c41ebbf6229cfe8d24a44 (DEFAULT_TABLE table=default.Table_5cfbfaf89e6a4bdf94cef7019cf9c5e4 [id=5c35ac21e1964919acdfdc5e72e5b6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:41.617331 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c098dcb1c55c41ebbf6229cfe8d24a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:41.617666 339 raft_consensus.cc:357] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.617821 339 raft_consensus.cc:383] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.617864 339 raft_consensus.cc:738] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.618043 339 consensus_queue.cc:260] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.620219 339 ts_tablet_manager.cc:1428] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:41.620489 339 tablet_bootstrap.cc:492] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.621970 339 tablet_bootstrap.cc:654] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.575075 32631 tablet_service.cc:1468] Processing CreateTablet for tablet 48ea78fb01d646d681404009eab9b050 (DEFAULT_TABLE table=default.Table_6b3d1637c4c1443089f04128e190256f [id=3f48141767f441dbbdf5a7742a16176f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:41.623035 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48ea78fb01d646d681404009eab9b050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:41.628304 32519 raft_consensus.cc:2466] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:41.615216 1854 raft_consensus.cc:491] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.615586 1903 raft_consensus.cc:491] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.629247 1903 raft_consensus.cc:513] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.628985 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_306d24f2f3484139b3586e0a1ebb8562"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:41.629911 1903 leader_election.cc:290] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:41.630245 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_306d24f2f3484139b3586e0a1ebb8562 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.630230 1854 raft_consensus.cc:513] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.630941 1854 leader_election.cc:290] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.632304 32314 leader_election.cc:304] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:41.632727 1682 raft_consensus.cc:2802] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.632925 1682 raft_consensus.cc:695] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:41.633153 1682 consensus_queue.cc:237] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.638049 339 tablet_bootstrap.cc:492] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.638218 339 ts_tablet_manager.cc:1397] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.002s
I20250623 08:04:41.639096 339 raft_consensus.cc:357] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.639261 339 raft_consensus.cc:383] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.639315 339 raft_consensus.cc:738] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.639497 339 consensus_queue.cc:260] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.639628 1359 raft_consensus.cc:491] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:41.639791 1359 raft_consensus.cc:513] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } }
I20250623 08:04:41.640064 339 ts_tablet_manager.cc:1428] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:41.640316 339 tablet_bootstrap.cc:492] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.640550 1359 leader_election.cc:290] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 7351aa31800245659edc9837e27daffd (127.31.69.1:34877)
I20250623 08:04:41.641934 339 tablet_bootstrap.cc:654] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.647367 979 raft_consensus.cc:2270] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.647970 979 ts_tablet_manager.cc:1905] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.648239 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b81263a9ee9d4da1ac721c3b2bd21248" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.650966 1682 raft_consensus.cc:491] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:41.651014 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 192) took 6327 ms. Trace:
I20250623 08:04:41.651127 952 rpcz_store.cc:276] 0623 08:04:35.323294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.323333 (+ 39us) service_pool.cc:225] Handling call
0623 08:04:41.415758 (+6092425us) ts_tablet_manager.cc:1145] Deleting tablet f4a0b9402a264800a1ea39707c10da0c
0623 08:04:41.415772 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.530671 (+114899us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.533889 (+ 3218us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.650991 (+117102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6092221}
I20250623 08:04:41.651226 1682 raft_consensus.cc:513] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:41.651314 32453 leader_election.cc:343] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: Tablet not RUNNING: STOPPED
I20250623 08:04:41.651520 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 193) took 6327 ms. Trace:
W20250623 08:04:41.651654 32453 leader_election.cc:343] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: b81263a9ee9d4da1ac721c3b2bd21248
I20250623 08:04:41.651615 952 rpcz_store.cc:276] 0623 08:04:35.323685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.323726 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:41.651424 (+6327698us) ts_tablet_manager.cc:1145] Deleting tablet 726d80e05a344f27b5a289723cb5aed7
0623 08:04:41.651435 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.651505 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6327508}
I20250623 08:04:41.651764 32453 leader_election.cc:304] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.651926 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 195) took 6327 ms. Trace:
I20250623 08:04:41.651976 1359 raft_consensus.cc:2747] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:41.651998 952 rpcz_store.cc:276] 0623 08:04:35.324399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.324439 (+ 40us) service_pool.cc:225] Handling call
0623 08:04:41.651854 (+6327415us) ts_tablet_manager.cc:1145] Deleting tablet b980c746a3c042bfae90aa076acbb1df
0623 08:04:41.651862 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.651912 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6327232}
I20250623 08:04:41.652259 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 196) took 6327 ms. Trace:
I20250623 08:04:41.652336 952 rpcz_store.cc:276] 0623 08:04:35.324772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.324813 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:41.652189 (+6327376us) ts_tablet_manager.cc:1145] Deleting tablet 32c82141edea4068b54755a85c6d7fb8
0623 08:04:41.652197 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.652246 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6327190}
I20250623 08:04:41.652660 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 197) took 6327 ms. Trace:
I20250623 08:04:41.652743 952 rpcz_store.cc:276] 0623 08:04:35.325176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.325218 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:41.652581 (+6327363us) ts_tablet_manager.cc:1145] Deleting tablet d4221d7500d1458db75926d7b274a654
0623 08:04:41.652591 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.652647 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6327176}
I20250623 08:04:41.653008 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 198) took 6327 ms. Trace:
I20250623 08:04:41.653091 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet f4a0b9402a264800a1ea39707c10da0c (table default.Table_244bcdc8d69b4f61a337dac403ffe6cb [id=b3ab55b66d754642aed2f746d9a35807]) successfully deleted
I20250623 08:04:41.653071 952 rpcz_store.cc:276] 0623 08:04:35.325557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.325597 (+ 40us) service_pool.cc:225] Handling call
0623 08:04:41.652938 (+6327341us) ts_tablet_manager.cc:1145] Deleting tablet 0311ce67989f49e8866234a88e5b4d09
0623 08:04:41.652946 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.652995 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6327172}
I20250623 08:04:41.653302 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 199) took 6327 ms. Trace:
W20250623 08:04:41.653340 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 726d80e05a344f27b5a289723cb5aed7 because the tablet was not found. No further retry: Not found: Tablet not found: 726d80e05a344f27b5a289723cb5aed7
W20250623 08:04:41.653621 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet b980c746a3c042bfae90aa076acbb1df because the tablet was not found. No further retry: Not found: Tablet not found: b980c746a3c042bfae90aa076acbb1df
I20250623 08:04:41.654124 339 tablet_bootstrap.cc:492] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.654230 339 ts_tablet_manager.cc:1397] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20250623 08:04:41.655064 339 raft_consensus.cc:357] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.655202 339 raft_consensus.cc:383] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.655242 339 raft_consensus.cc:738] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.655418 339 consensus_queue.cc:260] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.655915 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e17e1ca404b462da43aa1db7dc3b556" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.655982 339 ts_tablet_manager.cc:1428] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:41.656214 339 tablet_bootstrap.cc:492] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.656668 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16bc969d09f74b0986070a504f9f670e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.653367 952 rpcz_store.cc:276] 0623 08:04:35.325922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.325962 (+ 40us) service_pool.cc:225] Handling call
0623 08:04:41.653237 (+6327275us) ts_tablet_manager.cc:1145] Deleting tablet 67c44697ecab4dd9b43f544a602897ca
0623 08:04:41.653245 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.653290 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6327132}
I20250623 08:04:41.657647 952 tablet_replica.cc:331] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: stopping tablet replica
W20250623 08:04:41.657670 32314 leader_election.cc:343] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 1a070895f4984493a3b9f6fa779797a9
I20250623 08:04:41.657826 952 raft_consensus.cc:2241] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
W20250623 08:04:41.657889 32589 leader_election.cc:343] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.657953 1682 leader_election.cc:290] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.658102 339 tablet_bootstrap.cc:654] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:41.658080 32589 leader_election.cc:343] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.658152 952 raft_consensus.cc:2270] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.658728 952 ts_tablet_manager.cc:1905] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:41.659236 32045 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 16bc969d09f74b0986070a504f9f670e consensus state has no leader
W20250623 08:04:41.659780 32314 leader_election.cc:343] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 1a070895f4984493a3b9f6fa779797a9
I20250623 08:04:41.660101 32314 leader_election.cc:304] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:41.660580 1682 raft_consensus.cc:2747] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:41.661855 32054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_1e293608b61743279ff20716c2953e24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:41.662731 32054 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e293608b61743279ff20716c2953e24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.669196 1682 raft_consensus.cc:491] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.671844 1682 raft_consensus.cc:513] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.674057 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b5233522f43c7aed3c9770adb5c03" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:41.674274 32656 raft_consensus.cc:2391] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:41.675398 1682 leader_election.cc:290] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.675776 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b5233522f43c7aed3c9770adb5c03" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.687091 342 tablet_bootstrap.cc:492] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.687233 342 ts_tablet_manager.cc:1397] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.002s
I20250623 08:04:41.688120 342 raft_consensus.cc:357] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.688277 342 raft_consensus.cc:383] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.688328 342 raft_consensus.cc:738] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.688514 342 consensus_queue.cc:260] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.689090 342 ts_tablet_manager.cc:1428] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:41.689350 342 tablet_bootstrap.cc:492] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.693645 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e17e1ca404b462da43aa1db7dc3b556" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.693773 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16bc969d09f74b0986070a504f9f670e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:41.694311 342 tablet_bootstrap.cc:654] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:41.694657 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 32c82141edea4068b54755a85c6d7fb8 because the tablet was not found. No further retry: Not found: Tablet not found: 32c82141edea4068b54755a85c6d7fb8
W20250623 08:04:41.694995 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet d4221d7500d1458db75926d7b274a654 because the tablet was not found. No further retry: Not found: Tablet not found: d4221d7500d1458db75926d7b274a654
W20250623 08:04:41.695225 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 0311ce67989f49e8866234a88e5b4d09 because the tablet was not found. No further retry: Not found: Tablet not found: 0311ce67989f49e8866234a88e5b4d09
W20250623 08:04:41.695470 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 67c44697ecab4dd9b43f544a602897ca because the tablet was not found. No further retry: Not found: Tablet not found: 67c44697ecab4dd9b43f544a602897ca
I20250623 08:04:41.695794 32314 leader_election.cc:304] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:41.698068 1843 raft_consensus.cc:3058] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.700631 1843 raft_consensus.cc:2747] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.712826 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:41.713075 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:41.713651 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:41.714051 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:41.714339 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:41.714607 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:41.721078 952 ts_tablet_manager.cc:1918] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.721666 981 ts_tablet_manager.cc:1918] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.726183 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:41.726536 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (table default.Table_b764664d1bdc4441907690b6616f3570 [id=ad7210d2665f463c8591e79cf62d15dc]). Expected version 1 got 0
I20250623 08:04:41.727211 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:41.722060 342 tablet_bootstrap.cc:492] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.727522 342 ts_tablet_manager.cc:1397] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20250623 08:04:41.729251 342 raft_consensus.cc:357] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.729419 342 raft_consensus.cc:383] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.729545 342 raft_consensus.cc:738] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.729774 342 consensus_queue.cc:260] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.730016 981 log.cc:1199] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b81263a9ee9d4da1ac721c3b2bd21248
I20250623 08:04:41.730259 342 ts_tablet_manager.cc:1428] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:41.730576 342 tablet_bootstrap.cc:492] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.430135 343 tablet_bootstrap.cc:654] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.732790 342 tablet_bootstrap.cc:654] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:41.722676 32589 leader_election.cc:343] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.733013 32589 leader_election.cc:304] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:41.733656 32589 leader_election.cc:343] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.734344 952 log.cc:1199] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9a0bbafe1ea34ab5ae1b3de25b50d4a3
I20250623 08:04:41.734826 952 ts_tablet_manager.cc:1939] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:41.735159 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 200) took 6408 ms. Trace:
I20250623 08:04:41.735275 952 rpcz_store.cc:276] 0623 08:04:35.326291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.326329 (+ 38us) service_pool.cc:225] Handling call
0623 08:04:41.657604 (+6331275us) ts_tablet_manager.cc:1145] Deleting tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3
0623 08:04:41.657612 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.699475 (+ 41863us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.720967 (+ 21492us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.735145 (+ 14178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6331106}
I20250623 08:04:41.735821 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 201) took 6409 ms. Trace:
I20250623 08:04:41.735962 952 rpcz_store.cc:276] 0623 08:04:35.326661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.326699 (+ 38us) service_pool.cc:225] Handling call
0623 08:04:41.735698 (+6408999us) ts_tablet_manager.cc:1145] Deleting tablet 8dc696236a15420e9666cca708068fb2
0623 08:04:41.735706 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.735805 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6408809}
I20250623 08:04:41.736316 952 tablet_replica.cc:331] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:41.736553 952 raft_consensus.cc:2241] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:41.736884 952 pending_rounds.cc:70] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:41.736989 952 pending_rounds.cc:77] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727449390592000 op_type: NO_OP noop_request { }
I20250623 08:04:41.737160 952 raft_consensus.cc:2887] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:41.737303 952 raft_consensus.cc:2270] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.738315 952 ts_tablet_manager.cc:1905] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.743399 981 ts_tablet_manager.cc:1939] T b81263a9ee9d4da1ac721c3b2bd21248 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:41.743822 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:41.744191 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 246) took 5160 ms. Trace:
I20250623 08:04:41.744290 981 rpcz_store.cc:276] 0623 08:04:36.583450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.583608 (+ 158us) service_pool.cc:225] Handling call
0623 08:04:41.311297 (+4727689us) ts_tablet_manager.cc:1145] Deleting tablet b81263a9ee9d4da1ac721c3b2bd21248
0623 08:04:41.311305 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.324660 (+ 13355us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.721563 (+396903us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.744170 (+ 22607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4727497}
I20250623 08:04:41.744653 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 251) took 4937 ms. Trace:
I20250623 08:04:41.744742 981 rpcz_store.cc:276] 0623 08:04:36.807157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.807215 (+ 58us) service_pool.cc:225] Handling call
0623 08:04:41.744559 (+4937344us) ts_tablet_manager.cc:1145] Deleting tablet 228375592aee48a1aa21e1d16962b663
0623 08:04:41.744569 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.744639 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4931977}
I20250623 08:04:41.744624 32059 catalog_manager.cc:5582] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:41.747860 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f8a5e35c1974456b8079ab8907c2652e (table default.Table_d2473e1e63c84b229243a1c676266ca6 [id=c63ada5e02d74e049460990450ab4962]). Expected version 1 got 0
I20250623 08:04:41.734124 1877 raft_consensus.cc:2747] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.749209 32589 leader_election.cc:304] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.749505 1877 raft_consensus.cc:2747] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.746999 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet b81263a9ee9d4da1ac721c3b2bd21248 (table default.Table_4e377b569121400d82096403060f9cda [id=49bfa95de97b443ba51225f84820fcf2]) successfully deleted
W20250623 08:04:41.754397 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 228375592aee48a1aa21e1d16962b663 because the tablet was not found. No further retry: Not found: Tablet not found: 228375592aee48a1aa21e1d16962b663
I20250623 08:04:41.748297 32059 catalog_manager.cc:5582] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:41.755270 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:41.755725 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:41.759402 343 tablet_bootstrap.cc:492] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:41.762874 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 (table default.Table_e7f8192c00524b5899152fa3f717b898 [id=f4277b270809454da8cfd8f3158ed6f8]) successfully deleted
I20250623 08:04:41.763037 339 tablet_bootstrap.cc:492] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.763185 339 ts_tablet_manager.cc:1397] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.002s
W20250623 08:04:41.763206 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 8dc696236a15420e9666cca708068fb2 because the tablet was not found. No further retry: Not found: Tablet not found: 8dc696236a15420e9666cca708068fb2
I20250623 08:04:41.763887 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:41.764453 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:41.765663 1957 consensus_queue.cc:1035] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.772019 342 tablet_bootstrap.cc:492] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.772145 342 ts_tablet_manager.cc:1397] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20250623 08:04:41.768487 339 raft_consensus.cc:357] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.772691 339 raft_consensus.cc:383] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.772756 339 raft_consensus.cc:738] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.775250 339 consensus_queue.cc:260] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.777783 342 raft_consensus.cc:357] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.777949 342 raft_consensus.cc:383] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.777999 342 raft_consensus.cc:738] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.778151 342 consensus_queue.cc:260] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.778650 342 ts_tablet_manager.cc:1428] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250623 08:04:41.778940 342 tablet_bootstrap.cc:492] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.780355 339 ts_tablet_manager.cc:1428] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250623 08:04:41.780623 342 tablet_bootstrap.cc:654] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.776540 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:41.782001 343 ts_tablet_manager.cc:1397] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.354s user 0.002s sys 0.002s
I20250623 08:04:41.781639 1877 consensus_queue.cc:1035] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.782846 343 raft_consensus.cc:357] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.782989 343 raft_consensus.cc:383] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.783041 343 raft_consensus.cc:738] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.784560 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:41.784801 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:41.785104 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:41.785290 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:41.785421 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:41.787581 339 tablet_bootstrap.cc:492] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.789181 339 tablet_bootstrap.cc:654] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.785944 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:41.795022 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:41.796129 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 252) took 4988 ms. Trace:
I20250623 08:04:41.796252 981 rpcz_store.cc:276] 0623 08:04:36.807564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.807610 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:41.744955 (+4937345us) ts_tablet_manager.cc:1145] Deleting tablet 8d8f38a6de1e45e7bed4abdced684c92
0623 08:04:41.744964 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.796098 (+ 51134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4932320}
I20250623 08:04:41.795769 32059 catalog_manager.cc:5582] T e602d391526d4b0eb0dbf5189afbe075 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
W20250623 08:04:41.797114 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 8d8f38a6de1e45e7bed4abdced684c92 because the tablet was not found. No further retry: Not found: Tablet not found: 8d8f38a6de1e45e7bed4abdced684c92
I20250623 08:04:41.797883 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:41.798588 32059 catalog_manager.cc:5582] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:41.799067 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 254) took 4990 ms. Trace:
I20250623 08:04:41.799105 32059 catalog_manager.cc:5582] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:41.799186 981 rpcz_store.cc:276] 0623 08:04:36.808304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.808801 (+ 497us) service_pool.cc:225] Handling call
0623 08:04:41.796631 (+4987830us) ts_tablet_manager.cc:1145] Deleting tablet 726d80e05a344f27b5a289723cb5aed7
0623 08:04:41.798961 (+ 2330us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.799047 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4979597}
I20250623 08:04:41.799470 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
W20250623 08:04:41.799836 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 726d80e05a344f27b5a289723cb5aed7 because the tablet was not found. No further retry: Not found: Tablet not found: 726d80e05a344f27b5a289723cb5aed7
I20250623 08:04:41.800319 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 255) took 4991 ms. Trace:
I20250623 08:04:41.800462 981 rpcz_store.cc:276] 0623 08:04:36.809084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.809138 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:41.800191 (+4991053us) ts_tablet_manager.cc:1145] Deleting tablet c0f87fd62f1545089d00757f8214593a
0623 08:04:41.800225 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.800300 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4983102}
W20250623 08:04:41.801120 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet c0f87fd62f1545089d00757f8214593a because the tablet was not found. No further retry: Not found: Tablet not found: c0f87fd62f1545089d00757f8214593a
I20250623 08:04:41.801574 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 256) took 4992 ms. Trace:
I20250623 08:04:41.802557 981 rpcz_store.cc:276] 0623 08:04:36.809395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.809440 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:41.801479 (+4992039us) ts_tablet_manager.cc:1145] Deleting tablet 4ea81642050141a9be84f151a5785abc
0623 08:04:41.801492 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.801555 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4984350}
I20250623 08:04:41.803637 342 tablet_bootstrap.cc:492] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.803783 342 ts_tablet_manager.cc:1397] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
W20250623 08:04:41.803978 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 4ea81642050141a9be84f151a5785abc because the tablet was not found. No further retry: Not found: Tablet not found: 4ea81642050141a9be84f151a5785abc
I20250623 08:04:41.804328 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 258) took 4994 ms. Trace:
I20250623 08:04:41.804546 342 raft_consensus.cc:357] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.804713 342 raft_consensus.cc:383] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.804773 342 raft_consensus.cc:738] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.804986 342 consensus_queue.cc:260] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.805253 32380 raft_consensus.cc:3058] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.805485 342 ts_tablet_manager.cc:1428] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:41.805738 342 tablet_bootstrap.cc:492] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.807394 343 consensus_queue.cc:260] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.809146 981 rpcz_store.cc:276] 0623 08:04:36.809643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.809939 (+ 296us) service_pool.cc:225] Handling call
0623 08:04:41.804256 (+4994317us) ts_tablet_manager.cc:1145] Deleting tablet 67c44697ecab4dd9b43f544a602897ca
0623 08:04:41.804264 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.804314 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4987025}
W20250623 08:04:41.811682 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 67c44697ecab4dd9b43f544a602897ca because the tablet was not found. No further retry: Not found: Tablet not found: 67c44697ecab4dd9b43f544a602897ca
I20250623 08:04:41.812996 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 263) took 5002 ms. Trace:
I20250623 08:04:41.813421 981 rpcz_store.cc:276] 0623 08:04:36.810313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.810356 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:41.812895 (+5002539us) ts_tablet_manager.cc:1145] Deleting tablet ce1f3ef3690c4837b95b105733884f1f
0623 08:04:41.812903 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.812980 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4990043}
W20250623 08:04:41.815476 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet ce1f3ef3690c4837b95b105733884f1f because the tablet was not found. No further retry: Not found: Tablet not found: ce1f3ef3690c4837b95b105733884f1f
I20250623 08:04:41.815822 981 tablet_replica.cc:331] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:41.815963 1956 consensus_queue.cc:1035] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.816936 981 raft_consensus.cc:2241] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:41.819624 952 ts_tablet_manager.cc:1918] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.819792 952 log.cc:1199] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/501b5233522f43c7aed3c9770adb5c03
I20250623 08:04:41.820216 952 ts_tablet_manager.cc:1939] T 501b5233522f43c7aed3c9770adb5c03 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:41.820572 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 202) took 6311 ms. Trace:
I20250623 08:04:41.820842 952 rpcz_store.cc:276] 0623 08:04:35.509591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.509839 (+ 248us) service_pool.cc:225] Handling call
0623 08:04:41.736270 (+6226431us) ts_tablet_manager.cc:1145] Deleting tablet 501b5233522f43c7aed3c9770adb5c03
0623 08:04:41.736279 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.788161 (+ 51882us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.819514 (+ 31353us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.820558 (+ 1044us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6226165}
I20250623 08:04:41.821208 952 tablet_replica.cc:331] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:41.821437 952 raft_consensus.cc:2241] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:41.821633 952 raft_consensus.cc:2270] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.822235 952 ts_tablet_manager.cc:1905] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.823546 981 raft_consensus.cc:2270] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.827313 981 ts_tablet_manager.cc:1905] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.829015 343 ts_tablet_manager.cc:1428] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20250623 08:04:41.829329 343 tablet_bootstrap.cc:492] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:41.831408 343 tablet_bootstrap.cc:654] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.837110 342 tablet_bootstrap.cc:654] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.838084 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:41.838488 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:41.844801 979 ts_tablet_manager.cc:1918] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:41.845160 979 log.cc:1199] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/aa398354a9e44633857a75c0bbba08b3
I20250623 08:04:41.838740 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:41.845649 979 ts_tablet_manager.cc:1939] T aa398354a9e44633857a75c0bbba08b3 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:41.845749 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 1c529e826fa84a919c018887620e4023 (table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]). Expected version 1 got 0
I20250623 08:04:41.845909 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:41.846098 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:41.847705 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_217651efa5dc4571b2a47cd3196c9e57 [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 1 got 0
I20250623 08:04:41.848074 1956 raft_consensus.cc:491] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.848261 1956 raft_consensus.cc:513] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.848904 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:41.849125 1956 leader_election.cc:290] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.849436 32059 catalog_manager.cc:5582] T 87648b11b10b45fc9cc8c998bbceae92 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:41.849892 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:41.854672 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306c4021b5d94a4996de07c222a8bfd8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:41.854866 32649 raft_consensus.cc:2466] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:41.855098 32453 consensus_peers.cc:487] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48da1e4759bb4fea87a45ad8a428b6e7. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:41.855360 32453 leader_election.cc:304] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:41.855579 1956 raft_consensus.cc:2802] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:41.855724 1956 raft_consensus.cc:491] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:41.855870 1956 raft_consensus.cc:3058] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.858832 1956 raft_consensus.cc:513] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.859588 1956 leader_election.cc:290] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.831548 339 tablet_bootstrap.cc:492] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.862948 339 ts_tablet_manager.cc:1397] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.002s
I20250623 08:04:41.863826 339 raft_consensus.cc:357] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.864009 339 raft_consensus.cc:383] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.864074 339 raft_consensus.cc:738] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.864279 339 consensus_queue.cc:260] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.864463 342 tablet_bootstrap.cc:492] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:41.864578 342 ts_tablet_manager.cc:1397] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20250623 08:04:41.864802 339 ts_tablet_manager.cc:1428] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:41.865065 339 tablet_bootstrap.cc:492] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.865391 1682 raft_consensus.cc:491] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.865339 342 raft_consensus.cc:357] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.865483 342 raft_consensus.cc:383] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.865532 342 raft_consensus.cc:738] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.865512 1682 raft_consensus.cc:513] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.865702 342 consensus_queue.cc:260] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.854408 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306c4021b5d94a4996de07c222a8bfd8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.866086 1682 leader_election.cc:290] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.866169 342 ts_tablet_manager.cc:1428] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:41.866438 342 tablet_bootstrap.cc:492] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.867688 339 tablet_bootstrap.cc:654] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:41.867929 32314 leader_election.cc:343] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: b80a91ae2c084020b5d3444b570b28f2
I20250623 08:04:41.868098 342 tablet_bootstrap.cc:654] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:41.868148 32314 leader_election.cc:343] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: b80a91ae2c084020b5d3444b570b28f2
I20250623 08:04:41.868249 32314 leader_election.cc:304] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:41.868448 1682 raft_consensus.cc:2747] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:41.869035 1863 consensus_queue.cc:1035] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:41.869470 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet aa398354a9e44633857a75c0bbba08b3 (table default.Table_6bb7361aabf7440589a3958da502f36c [id=150ddf7cf1b141bc8586b94cf6f34aba]) successfully deleted
W20250623 08:04:41.869850 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 52d0a356c74f4daea9093c998b4bf73e because the tablet was not found. No further retry: Not found: Tablet not found: 52d0a356c74f4daea9093c998b4bf73e
I20250623 08:04:41.870143 979 tablet_replica.cc:331] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
W20250623 08:04:41.870162 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 3249605158634f4eab5b9d61357c383f because the tablet was not found. No further retry: Not found: Tablet not found: 3249605158634f4eab5b9d61357c383f
I20250623 08:04:41.870404 979 raft_consensus.cc:2241] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:41.870450 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 724d662bcf754c74a8f3dff91ee258ae because the tablet was not found. No further retry: Not found: Tablet not found: 724d662bcf754c74a8f3dff91ee258ae
I20250623 08:04:41.870609 979 raft_consensus.cc:2270] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:41.870729 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 283bebc7063d437ba3e541d7dee3c0b5 because the tablet was not found. No further retry: Not found: Tablet not found: 283bebc7063d437ba3e541d7dee3c0b5
I20250623 08:04:41.876415 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306c4021b5d94a4996de07c222a8bfd8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:41.877955 32453 leader_election.cc:343] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.877684 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306c4021b5d94a4996de07c222a8bfd8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:41.878192 32642 raft_consensus.cc:3058] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.882959 32642 raft_consensus.cc:2466] T 306c4021b5d94a4996de07c222a8bfd8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:41.886687 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:41.887066 32059 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:41.887096 979 ts_tablet_manager.cc:1905] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:41.888082 32050 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet baccec88aefb4056be9578bca13f1591 consensus state has no leader
I20250623 08:04:41.888259 32050 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 402) took 1064 ms. Trace:
W20250623 08:04:41.888375 32056 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1cee3de5953244f98428747bea912f77 consensus state has no leader
I20250623 08:04:41.888582 32056 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 403) took 1060 ms. Trace:
W20250623 08:04:41.888612 32057 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88f2a63b5a734c8f9b7a53361590245d consensus state has no leader
I20250623 08:04:41.888760 32057 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36116 (request call id 405) took 1019 ms. Trace:
I20250623 08:04:41.888914 32057 rpcz_store.cc:276] 0623 08:04:40.869170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.508556 (+639386us) service_pool.cc:225] Handling call
0623 08:04:41.510536 (+ 1980us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07fb5a6 00007f19c7646616 00007f19c7646345 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:41.510877 (+ 341us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:41.511082 (+ 205us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:41.511246 (+ 164us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:41.511325 (+ 79us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:41.577943 (+ 66618us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:41.888720 (+310777us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.516804 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.516901 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.516906 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:41.516909 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:41.517002 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.517011 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.517016 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:41.517083 (+ 67us) tablet.cc:699] Row locks acquired
0623 08:04:41.517086 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:41.517114 (+ 28us) write_op.cc:270] Start()
0623 08:04:41.517139 (+ 25us) write_op.cc:276] Timestamp: P: 1750665881517075 usec, L: 0
0623 08:04:41.517142 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.517283 (+ 141us) log.cc:844] Serialized 405 byte log entry
0623 08:04:41.576971 (+ 59688us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.577549 (+ 578us) write_op.cc:301] APPLY: starting
0623 08:04:41.577564 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.577596 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.577621 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.577660 (+ 39us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.577687 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:41.577694 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:41.577752 (+ 58us) log.cc:844] Serialized 23 byte log entry
0623 08:04:41.577804 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.577818 (+ 14us) write_op.cc:454] Released schema lock
0623 08:04:41.577823 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":3850496,"child_traces":[["op",{"apply.queue_time_us":521,"apply.run_cpu_time_us":336,"apply.run_wall_time_us":335,"num_ops":1,"prepare.queue_time_us":5047,"prepare.run_cpu_time_us":604,"prepare.run_wall_time_us":25984,"raft.queue_time_us":32,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":208,"replication_time_us":59799}]]}
I20250623 08:04:41.888376 32050 rpcz_store.cc:276] 0623 08:04:40.823901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.467011 (+643110us) service_pool.cc:225] Handling call
0623 08:04:41.467430 (+ 419us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:41.467643 (+ 213us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:41.467845 (+ 202us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:41.467959 (+ 114us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:41.547891 (+ 79932us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:41.888216 (+340325us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.479185 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.479295 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.479302 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:41.479304 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:41.479405 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.479414 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.479418 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:41.479489 (+ 71us) tablet.cc:699] Row locks acquired
0623 08:04:41.479491 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:41.479525 (+ 34us) write_op.cc:270] Start()
0623 08:04:41.479552 (+ 27us) write_op.cc:276] Timestamp: P: 1750665881479490 usec, L: 0
0623 08:04:41.479555 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.479687 (+ 132us) log.cc:844] Serialized 405 byte log entry
0623 08:04:41.547169 (+ 67482us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.547449 (+ 280us) write_op.cc:301] APPLY: starting
0623 08:04:41.547474 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.547513 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.547547 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.547593 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.547625 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:41.547633 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:41.547702 (+ 69us) log.cc:844] Serialized 23 byte log entry
0623 08:04:41.547755 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.547771 (+ 16us) write_op.cc:454] Released schema lock
0623 08:04:41.547777 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":229,"apply.run_cpu_time_us":383,"apply.run_wall_time_us":383,"num_ops":1,"prepare.queue_time_us":10648,"prepare.run_cpu_time_us":591,"prepare.run_wall_time_us":4368,"raft.queue_time_us":872,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":211,"replication_time_us":67580}]]}
I20250623 08:04:41.891489 32050 master_service.cc:948] ReplaceTablet: received request to replace tablet f9f4829e103b460eb582beff3b966bb4 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:41.888674 32056 rpcz_store.cc:276] 0623 08:04:40.827684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.469267 (+641583us) service_pool.cc:225] Handling call
0623 08:04:41.469707 (+ 440us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:41.469977 (+ 270us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:41.470171 (+ 194us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:41.470244 (+ 73us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:41.555137 (+ 84893us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:41.888509 (+333372us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.491785 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.491903 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.491909 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:41.491911 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:41.492003 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.492015 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.492021 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:41.492074 (+ 53us) tablet.cc:699] Row locks acquired
0623 08:04:41.492077 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:41.492106 (+ 29us) write_op.cc:270] Start()
0623 08:04:41.492133 (+ 27us) write_op.cc:276] Timestamp: P: 1750665881492069 usec, L: 0
0623 08:04:41.492138 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.492308 (+ 170us) log.cc:844] Serialized 405 byte log entry
0623 08:04:41.551788 (+ 59480us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.552755 (+ 967us) write_op.cc:301] APPLY: starting
0623 08:04:41.552772 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.552808 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.552835 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.552877 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.552913 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:41.552923 (+ 10us) write_op.cc:312] APPLY: finished
0623 08:04:41.552989 (+ 66us) log.cc:844] Serialized 23 byte log entry
0623 08:04:41.553048 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.553066 (+ 18us) write_op.cc:454] Released schema lock
0623 08:04:41.553074 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":914,"apply.run_cpu_time_us":389,"apply.run_wall_time_us":389,"num_ops":1,"prepare.queue_time_us":21113,"prepare.run_cpu_time_us":609,"prepare.run_wall_time_us":3253,"raft.queue_time_us":1008,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":198,"replication_time_us":59622}]]}
I20250623 08:04:41.892179 32056 master_service.cc:948] ReplaceTablet: received request to replace tablet ba5913811e9343389c9ea0ec8b8c05ef from {username='slave'} at 127.0.0.1:36116
W20250623 08:04:41.890691 32055 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3bfad68d1f20486997ea1b0a8d97206b consensus state has no leader
W20250623 08:04:41.894244 32055 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e41b15c89a744a698ca5eee2f143b4bc consensus state has no leader
I20250623 08:04:41.894636 981 ts_tablet_manager.cc:1918] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250623 08:04:41.894712 32055 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a375f8ea383b44a9acd04fc2171d34ae consensus state has no leader
I20250623 08:04:41.895344 339 tablet_bootstrap.cc:492] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.895471 339 ts_tablet_manager.cc:1397] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20250623 08:04:41.895988 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 16331ba6a34942a5a1b4ff4ac7bac55e with delete_type TABLET_DATA_DELETED (replaced by tablet 4e3920ac85e54379abbcc42dd80db504) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.896538 339 raft_consensus.cc:357] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.896797 339 raft_consensus.cc:383] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.896903 339 raft_consensus.cc:738] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.897152 339 consensus_queue.cc:260] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.897670 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.897858 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet b84b8b8713774ab4896842feb9fb075e with delete_type TABLET_DATA_DELETED (replaced by tablet a3576624314d4e85b411aab9109683f7) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.897998 32349 tablet_service.cc:1515] Processing DeleteTablet for tablet b61b02cf2904404f8c0ffef7d268fefc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.898059 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.898017 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 36c8714cfbd84dd38cf5c0f305ef6843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:34 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.898187 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f62ec83c88e43438b963e740d2c7004 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.898293 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet 18ea63263b9b466e922169eecafab747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.898392 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 501b5233522f43c7aed3c9770adb5c03 with delete_type TABLET_DATA_DELETED (replaced by tablet 74c1cdc83ffd4a09a163479ac58fd405) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.898432 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet b81263a9ee9d4da1ac721c3b2bd21248 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.898603 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet f5f58f55070344ff98a3808f70e0c755 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.899102 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet b720fdea539f423ba2d2aa9f89b09ae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.899518 1682 raft_consensus.cc:491] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.899631 1682 raft_consensus.cc:513] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.900199 1682 leader_election.cc:290] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.900651 339 ts_tablet_manager.cc:1428] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:41.900974 339 tablet_bootstrap.cc:492] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:41.901198 32314 leader_election.cc:343] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 36c8714cfbd84dd38cf5c0f305ef6843
I20250623 08:04:41.902741 339 tablet_bootstrap.cc:654] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.904019 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36c8714cfbd84dd38cf5c0f305ef6843" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:41.905205 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet 26f4a1b472e848ccb9a1ec71c1dd3516 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:35542
W20250623 08:04:41.908846 32314 leader_election.cc:343] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: Tablet not RUNNING: STOPPED
W20250623 08:04:41.909390 32025 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 67974.3us
W20250623 08:04:41.913045 32453 leader_election.cc:343] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.913710 32050 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_8ef30265c1894796843272f538aaa79e" } modify_external_catalogs: true
I20250623 08:04:41.913920 32050 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_8ef30265c1894796843272f538aaa79e" } modify_external_catalogs: true
I20250623 08:04:41.915792 32453 leader_election.cc:304] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.916069 1823 raft_consensus.cc:2802] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:41.916181 1823 raft_consensus.cc:695] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:41.916450 1823 consensus_queue.cc:237] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.892853 952 ts_tablet_manager.cc:1918] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:41.916774 952 log.cc:1199] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/789d1c20aa72480ab71123c799bbde4c
I20250623 08:04:41.917214 952 ts_tablet_manager.cc:1939] T 789d1c20aa72480ab71123c799bbde4c P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:41.917973 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 203) took 6408 ms. Trace:
I20250623 08:04:41.918047 952 rpcz_store.cc:276] 0623 08:04:35.509775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.510153 (+ 378us) service_pool.cc:225] Handling call
0623 08:04:41.821165 (+6311012us) ts_tablet_manager.cc:1145] Deleting tablet 789d1c20aa72480ab71123c799bbde4c
0623 08:04:41.821173 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.825529 (+ 4356us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.892755 (+ 67226us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.917955 (+ 25200us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6310814}
I20250623 08:04:41.918440 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 204) took 6396 ms. Trace:
I20250623 08:04:41.918510 952 rpcz_store.cc:276] 0623 08:04:35.522261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:35.522308 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:41.918330 (+6396022us) ts_tablet_manager.cc:1145] Deleting tablet db81250d1e164a7d94affd764fc0ff0c
0623 08:04:41.918339 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.918425 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6395799}
I20250623 08:04:41.918870 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 208) took 5911 ms. Trace:
W20250623 08:04:41.921885 32589 consensus_peers.cc:487] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:41.922112 32056 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet ba5913811e9343389c9ea0ec8b8c05ef
W20250623 08:04:41.922221 32051 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:39501 (request call id 171) took 2035 ms (client timeout 1000 ms). Trace:
W20250623 08:04:41.922363 32051 rpcz_store.cc:269] 0623 08:04:39.886710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.898997 (+ 12287us) service_pool.cc:225] Handling call
0623 08:04:41.922168 (+2023171us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:40.660396 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:40.660525 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:40.660532 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:40.660535 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:40.660764 (+ 229us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:40.660781 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:40.660787 (+ 6us) tablet.cc:683] Acquiring locks for 4 operations
0623 08:04:40.660905 (+ 118us) tablet.cc:699] Row locks acquired
0623 08:04:40.660908 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:40.660947 (+ 39us) write_op.cc:270] Start()
0623 08:04:40.660983 (+ 36us) write_op.cc:276] Timestamp: P: 1750665880660899 usec, L: 0
0623 08:04:40.660987 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:40.661186 (+ 199us) log.cc:844] Serialized 1650 byte log entry
0623 08:04:40.682197 (+ 21011us) op_driver.cc:464] REPLICATION: finished
0623 08:04:40.682292 (+ 95us) write_op.cc:301] APPLY: starting
0623 08:04:40.682307 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:40.682397 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:40.682423 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:40.682517 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:40.682553 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0623 08:04:40.682560 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:40.682644 (+ 84us) log.cc:844] Serialized 41 byte log entry
0623 08:04:40.682706 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.682726 (+ 20us) write_op.cc:454] Released schema lock
0623 08:04:40.682733 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:40.682821 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:40.682826 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5868,"reported_tablets":208,"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":516,"apply.run_wall_time_us":516,"num_ops":4,"prepare.queue_time_us":290,"prepare.run_cpu_time_us":899,"prepare.run_wall_time_us":900,"raft.queue_time_us":6867,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":280,"replication_time_us":21174}]]}
I20250623 08:04:41.923950 32051 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_7202dd9a46994e949c2368ac8f51b53d" } modify_external_catalogs: true
I20250623 08:04:41.924405 32051 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_7202dd9a46994e949c2368ac8f51b53d" } modify_external_catalogs: true
I20250623 08:04:41.923870 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 92005abc61dd40ef9da8124cadc29b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.924269 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet cdec6e9573b041e0a57d1e0e0239d97f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:32 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.924803 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 92005abc61dd40ef9da8124cadc29b6a) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.925087 952 rpcz_store.cc:276] 0623 08:04:36.007771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.007826 (+ 55us) service_pool.cc:225] Handling call
0623 08:04:41.918733 (+5910907us) ts_tablet_manager.cc:1145] Deleting tablet d19d468ce13d4108a9badc36fc503306
0623 08:04:41.918741 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.918855 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5910722}
I20250623 08:04:41.925292 32484 tablet_service.cc:1515] Processing DeleteTablet for tablet 9049e3d9b39f420bb51221808409fc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.925618 32484 tablet_service.cc:1515] Processing DeleteTablet for tablet a414144276f6484ea4690204752e2142 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.925851 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet a25bdc9b7e48467fba3771365e58bcbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.926370 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 501b5233522f43c7aed3c9770adb5c03 with delete_type TABLET_DATA_DELETED (replaced by tablet 74c1cdc83ffd4a09a163479ac58fd405) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.926067 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet 965f1f0a107b4653b6aac01e0d8e259a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.930020 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:27 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.925532 952 tablet_replica.cc:331] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: stopping tablet replica
W20250623 08:04:41.930287 32589 consensus_peers.cc:487] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:41.930501 1863 raft_consensus.cc:491] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.930539 952 raft_consensus.cc:2241] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:41.930702 952 raft_consensus.cc:2270] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.930641 1863 raft_consensus.cc:513] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.931245 1863 leader_election.cc:290] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.931290 952 ts_tablet_manager.cc:1905] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.932055 343 tablet_bootstrap.cc:492] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:41.932204 343 ts_tablet_manager.cc:1397] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.000s
I20250623 08:04:41.932927 343 raft_consensus.cc:357] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.933075 343 raft_consensus.cc:383] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.933131 343 raft_consensus.cc:738] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.933332 343 consensus_queue.cc:260] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:41.937669 32050 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet adc3cf4355c24b7395fcb22af8520346
I20250623 08:04:41.914335 339 tablet_bootstrap.cc:492] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:41.940001 339 ts_tablet_manager.cc:1397] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250623 08:04:41.940786 339 raft_consensus.cc:357] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.940984 339 raft_consensus.cc:383] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:41.941045 339 raft_consensus.cc:738] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:41.941236 339 consensus_queue.cc:260] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.941838 339 ts_tablet_manager.cc:1428] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:41.942124 339 tablet_bootstrap.cc:492] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:41.943719 339 tablet_bootstrap.cc:654] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.944312 343 ts_tablet_manager.cc:1428] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250623 08:04:41.944641 343 tablet_bootstrap.cc:492] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:41.909036 32314 leader_election.cc:304] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:41.945456 1682 raft_consensus.cc:2747] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:41.946519 343 tablet_bootstrap.cc:654] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.948153 1784 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250623 08:04:41.955299 1956 raft_consensus.cc:491] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.955520 1956 raft_consensus.cc:513] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:41.956126 1956 leader_election.cc:290] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.931593 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8945b6f49474648ac73bcb85285680a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:41.928923 981 log.cc:1199] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/a414144276f6484ea4690204752e2142
I20250623 08:04:41.957595 981 ts_tablet_manager.cc:1939] T a414144276f6484ea4690204752e2142 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:41.958398 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 259) took 5148 ms. Trace:
I20250623 08:04:41.958537 981 rpcz_store.cc:276] 0623 08:04:36.809739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.810107 (+ 368us) service_pool.cc:225] Handling call
0623 08:04:41.815778 (+5005671us) ts_tablet_manager.cc:1145] Deleting tablet a414144276f6484ea4690204752e2142
0623 08:04:41.815787 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.891332 (+ 75545us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.894549 (+ 3217us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.958382 (+ 63833us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4988920}
I20250623 08:04:41.958983 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 264) took 5148 ms. Trace:
I20250623 08:04:41.959203 981 rpcz_store.cc:276] 0623 08:04:36.810645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.810689 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:41.958906 (+5148217us) ts_tablet_manager.cc:1145] Deleting tablet 35ab9f77a81a4490abb0d2c0c69cd90f
0623 08:04:41.958914 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.958970 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5131980}
I20250623 08:04:41.959636 981 tablet_replica.cc:331] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:41.959875 981 raft_consensus.cc:2241] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:41.960071 981 raft_consensus.cc:2270] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.960623 981 ts_tablet_manager.cc:1905] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.961843 952 ts_tablet_manager.cc:1918] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:41.962831 952 log.cc:1199] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/18ea63263b9b466e922169eecafab747
I20250623 08:04:41.963671 32056 catalog_manager.cc:6652] ReplaceTablet: tablet ba5913811e9343389c9ea0ec8b8c05ef deleted and replaced by tablet 2ec871b2878a4b3f929b3774746d7750
I20250623 08:04:41.964561 32056 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_c7119eddeaf44e1b89d2dbd1e38151f7" } new_table_name: "default.Table_7ccb5edd6b824f308d349c51d926d963" modify_external_catalogs: true
I20250623 08:04:41.965909 952 ts_tablet_manager.cc:1939] T 18ea63263b9b466e922169eecafab747 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:41.965874 32050 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_d74a4572f91e48d0b1c123e11ecf5b51" } new_table_name: "default.Table_a96d9f3fd33f485c8a9015f78f39a7e9" modify_external_catalogs: true
I20250623 08:04:41.966811 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 209) took 5958 ms. Trace:
I20250623 08:04:41.967017 952 rpcz_store.cc:276] 0623 08:04:36.008136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.009499 (+ 1363us) service_pool.cc:225] Handling call
0623 08:04:41.925486 (+5915987us) ts_tablet_manager.cc:1145] Deleting tablet 18ea63263b9b466e922169eecafab747
0623 08:04:41.925495 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.941621 (+ 16126us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.961738 (+ 20117us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.966790 (+ 5052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5915777}
I20250623 08:04:41.967433 952 tablet_replica.cc:331] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:41.968153 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet a414144276f6484ea4690204752e2142 (table default.Table_a4ca66fa08cc441ab5571f9968f30914 [id=c3c5e1869ba745b9b2ab29742806f241]) successfully deleted
W20250623 08:04:41.968426 32043 catalog_manager.cc:4895] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 35ab9f77a81a4490abb0d2c0c69cd90f because the tablet was not found. No further retry: Not found: Tablet not found: 35ab9f77a81a4490abb0d2c0c69cd90f
I20250623 08:04:41.968443 952 raft_consensus.cc:2241] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:41.968683 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 789d1c20aa72480ab71123c799bbde4c (table default.Table_da00c692e65345c496f4e52c106aa2e6 [id=5ea293756d514f06b8d749c615417969]) successfully deleted
W20250623 08:04:41.968901 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet db81250d1e164a7d94affd764fc0ff0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db81250d1e164a7d94affd764fc0ff0c already in progress: creating tablet
I20250623 08:04:41.969633 1823 raft_consensus.cc:491] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.969765 1823 raft_consensus.cc:513] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:41.970029 32055 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.1:45613 (request call id 158) took 2543 ms (client timeout 1000 ms). Trace:
W20250623 08:04:41.970160 32055 rpcz_store.cc:269] 0623 08:04:39.426998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.453735 (+ 26737us) service_pool.cc:225] Handling call
0623 08:04:39.563156 (+109421us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 00007f19cbe2bfcd 00007f19cbe2c0c3 00007f19c8af01c1 00007f19d06702ed 00007f19d07f3b58 00007f19c7640176 00007f19c763fea5 00007f19c3e3293e 00007f19c3e3181e 00007f19c3e37a55 00007f19c3e3837f
0623 08:04:41.969959 (+2406803us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":126089,"reported_tablets":213,"spinlock_wait_cycles":240817536}
I20250623 08:04:41.970335 1823 leader_election.cc:290] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:41.971283 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:55526
W20250623 08:04:41.970172 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet d19d468ce13d4108a9badc36fc503306 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d19d468ce13d4108a9badc36fc503306 already in progress: creating tablet
I20250623 08:04:41.971479 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.971477 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 36c8714cfbd84dd38cf5c0f305ef6843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:34 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.970712 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a0b9402a264800a1ea39707c10da0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.971637 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 18ea63263b9b466e922169eecafab747 (table default.Table_3540dde86f8442a89a59232aa4b53612 [id=6389457a4f384f9c9a69146391741b20]) successfully deleted
I20250623 08:04:41.971738 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 with delete_type TABLET_DATA_DELETED (replaced by tablet 306c4021b5d94a4996de07c222a8bfd8) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.972002 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet adc3cf4355c24b7395fcb22af8520346 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.972769 32484 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 077bb090b3ea4d029463304d7a6feee5) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.973011 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet ba5913811e9343389c9ea0ec8b8c05ef with delete_type TABLET_DATA_DELETED (replaced by tablet 2ec871b2878a4b3f929b3774746d7750) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:41.973017 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8945b6f49474648ac73bcb85285680a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:41.957041 32589 leader_election.cc:343] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.967574 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet ba5913811e9343389c9ea0ec8b8c05ef with delete_type TABLET_DATA_DELETED (replaced by tablet 2ec871b2878a4b3f929b3774746d7750) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:41.969133 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9f4829e103b460eb582beff3b966bb4" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:41.977973 32647 raft_consensus.cc:2466] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:41.969470 32631 tablet_service.cc:1515] Processing DeleteTablet for tablet adc3cf4355c24b7395fcb22af8520346 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:36612
W20250623 08:04:41.980727 32453 leader_election.cc:343] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 771ecfc41e2b4007a4d213cec5b128c6
I20250623 08:04:41.980998 32453 leader_election.cc:304] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:41.981254 1823 raft_consensus.cc:2802] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:41.981364 1823 raft_consensus.cc:491] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:41.981494 1823 raft_consensus.cc:3058] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:41.986158 952 raft_consensus.cc:2270] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:41.987294 952 ts_tablet_manager.cc:1905] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:41.991950 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_68f032ade7314d0da985fa8c94b11a8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:41.992560 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68f032ade7314d0da985fa8c94b11a8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:41.995683 979 ts_tablet_manager.cc:1918] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:41.995798 979 log.cc:1199] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/36c8714cfbd84dd38cf5c0f305ef6843
I20250623 08:04:41.996097 979 ts_tablet_manager.cc:1939] T 36c8714cfbd84dd38cf5c0f305ef6843 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:41.997656 32589 leader_election.cc:343] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:41.997783 32589 leader_election.cc:304] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:41.998134 1902 raft_consensus.cc:2747] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:41.999570 32589 consensus_peers.cc:487] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 789d1c20aa72480ab71123c799bbde4c. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:42.011900 32055 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e8b3bfa63ee7407ead7f4a0f8204d269" } modify_external_catalogs: true
I20250623 08:04:42.012113 32055 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_e8b3bfa63ee7407ead7f4a0f8204d269" } modify_external_catalogs: true
I20250623 08:04:42.016340 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 36c8714cfbd84dd38cf5c0f305ef6843 (table default.Table_2092a828aea549a38d1bd42248c8804c [id=cce6d993b26f4f9ca197d807107e00cb]) successfully deleted
W20250623 08:04:42.017992 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 because the tablet was not found. No further retry: Not found: Tablet not found: d0b2af698b6e4ed0ae66a9bb32be7b31
W20250623 08:04:42.018875 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 136e482995a24c5daec55a701f3091fa because the tablet was not found. No further retry: Not found: Tablet not found: 136e482995a24c5daec55a701f3091fa
I20250623 08:04:42.022142 952 ts_tablet_manager.cc:1918] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250623 08:04:42.023370 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f4a96ae9190640dea11bd11a9f0e4372 because the tablet was not found. No further retry: Not found: Tablet not found: f4a96ae9190640dea11bd11a9f0e4372
I20250623 08:04:42.023612 952 log.cc:1199] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/26f4a1b472e848ccb9a1ec71c1dd3516
I20250623 08:04:42.023938 979 tablet_replica.cc:331] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:42.024044 952 ts_tablet_manager.cc:1939] T 26f4a1b472e848ccb9a1ec71c1dd3516 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:42.024438 979 raft_consensus.cc:2241] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:42.024842 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 210) took 5979 ms. Trace:
I20250623 08:04:42.026826 979 raft_consensus.cc:2270] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.026988 952 rpcz_store.cc:276] 0623 08:04:36.045982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.046249 (+ 267us) service_pool.cc:225] Handling call
0623 08:04:41.967385 (+5921136us) ts_tablet_manager.cc:1145] Deleting tablet 26f4a1b472e848ccb9a1ec71c1dd3516
0623 08:04:41.967396 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.018946 (+ 51550us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.022052 (+ 3106us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.024821 (+ 2769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5920321}
I20250623 08:04:42.027465 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 211) took 5979 ms. Trace:
I20250623 08:04:42.027594 952 rpcz_store.cc:276] 0623 08:04:36.049002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.049413 (+ 411us) service_pool.cc:225] Handling call
0623 08:04:42.027345 (+5977932us) ts_tablet_manager.cc:1145] Deleting tablet 89bb7ea31b4d4ce7a3c063853142fbb2
0623 08:04:42.027354 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.027449 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5971015}
I20250623 08:04:42.027976 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 212) took 5888 ms. Trace:
I20250623 08:04:42.028098 952 rpcz_store.cc:276] 0623 08:04:36.139522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.139591 (+ 69us) service_pool.cc:225] Handling call
0623 08:04:42.027897 (+5888306us) ts_tablet_manager.cc:1145] Deleting tablet 776b412b3f564a7581c6a5e1c84e8499
0623 08:04:42.027905 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.027960 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5888125}
I20250623 08:04:42.028443 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 213) took 5888 ms. Trace:
I20250623 08:04:42.028571 952 rpcz_store.cc:276] 0623 08:04:36.140058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.140112 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:42.028367 (+5888255us) ts_tablet_manager.cc:1145] Deleting tablet 8d8f38a6de1e45e7bed4abdced684c92
0623 08:04:42.028377 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.028429 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5887915}
I20250623 08:04:42.028919 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 214) took 5888 ms. Trace:
I20250623 08:04:42.029043 952 rpcz_store.cc:276] 0623 08:04:36.140652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.140701 (+ 49us) service_pool.cc:225] Handling call
0623 08:04:42.028851 (+5888150us) ts_tablet_manager.cc:1145] Deleting tablet 35ab9f77a81a4490abb0d2c0c69cd90f
0623 08:04:42.028859 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.028906 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5887968}
I20250623 08:04:42.033886 979 ts_tablet_manager.cc:1905] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.031515 342 tablet_bootstrap.cc:492] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.034165 342 ts_tablet_manager.cc:1397] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.168s user 0.002s sys 0.017s
I20250623 08:04:42.035199 342 raft_consensus.cc:357] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.035455 342 raft_consensus.cc:383] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.035576 342 raft_consensus.cc:738] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.035830 342 consensus_queue.cc:260] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.036474 342 ts_tablet_manager.cc:1428] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:42.036787 342 tablet_bootstrap.cc:492] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.038282 342 tablet_bootstrap.cc:654] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.039203 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 215) took 5898 ms. Trace:
I20250623 08:04:42.039336 952 rpcz_store.cc:276] 0623 08:04:36.141113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.141155 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:42.039114 (+5897959us) ts_tablet_manager.cc:1145] Deleting tablet 4ea81642050141a9be84f151a5785abc
0623 08:04:42.039122 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.039188 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5897802}
I20250623 08:04:42.040078 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 217) took 5898 ms. Trace:
I20250623 08:04:42.040210 952 rpcz_store.cc:276] 0623 08:04:36.141874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.141912 (+ 38us) service_pool.cc:225] Handling call
0623 08:04:42.040003 (+5898091us) ts_tablet_manager.cc:1145] Deleting tablet 162308da898d4dde8ab5b1ab01d0f393
0623 08:04:42.040011 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.040063 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5897965}
I20250623 08:04:42.040553 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 218) took 5898 ms. Trace:
I20250623 08:04:42.040685 952 rpcz_store.cc:276] 0623 08:04:36.142155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.142222 (+ 67us) service_pool.cc:225] Handling call
0623 08:04:42.040485 (+5898263us) ts_tablet_manager.cc:1145] Deleting tablet 205da880685b49b3b8669f73dc2c563b
0623 08:04:42.040493 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.040540 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5898099}
I20250623 08:04:42.041019 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 219) took 5898 ms. Trace:
I20250623 08:04:42.041137 952 rpcz_store.cc:276] 0623 08:04:36.142617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.142669 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:42.040951 (+5898282us) ts_tablet_manager.cc:1145] Deleting tablet 9918b15e48144ef0b8f3aedc2135be59
0623 08:04:42.040959 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.041007 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5898131}
I20250623 08:04:42.041458 952 tablet_replica.cc:331] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:42.041709 952 raft_consensus.cc:2241] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:42.042024 952 raft_consensus.cc:2270] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.042253 1981 tablet.cc:1722] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:42.042819 952 ts_tablet_manager.cc:1905] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.044028 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 26f4a1b472e848ccb9a1ec71c1dd3516 (table default.Table_61ccd9db16cb4618874bff73413fd485 [id=8d6f6785bcf24b719419015368a05f68]) successfully deleted
W20250623 08:04:42.044430 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 89bb7ea31b4d4ce7a3c063853142fbb2 already in progress: creating tablet
W20250623 08:04:42.044798 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 776b412b3f564a7581c6a5e1c84e8499 because the tablet was not found. No further retry: Not found: Tablet not found: 776b412b3f564a7581c6a5e1c84e8499
W20250623 08:04:42.045116 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 8d8f38a6de1e45e7bed4abdced684c92 because the tablet was not found. No further retry: Not found: Tablet not found: 8d8f38a6de1e45e7bed4abdced684c92
W20250623 08:04:42.045435 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 35ab9f77a81a4490abb0d2c0c69cd90f because the tablet was not found. No further retry: Not found: Tablet not found: 35ab9f77a81a4490abb0d2c0c69cd90f
W20250623 08:04:42.045758 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 4ea81642050141a9be84f151a5785abc because the tablet was not found. No further retry: Not found: Tablet not found: 4ea81642050141a9be84f151a5785abc
W20250623 08:04:42.046154 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 162308da898d4dde8ab5b1ab01d0f393 because the tablet was not found. No further retry: Not found: Tablet not found: 162308da898d4dde8ab5b1ab01d0f393
W20250623 08:04:42.046444 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 205da880685b49b3b8669f73dc2c563b because the tablet was not found. No further retry: Not found: Tablet not found: 205da880685b49b3b8669f73dc2c563b
W20250623 08:04:42.046828 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 9918b15e48144ef0b8f3aedc2135be59 because the tablet was not found. No further retry: Not found: Tablet not found: 9918b15e48144ef0b8f3aedc2135be59
I20250623 08:04:42.056785 342 tablet_bootstrap.cc:492] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.057039 342 ts_tablet_manager.cc:1397] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
I20250623 08:04:42.057902 342 raft_consensus.cc:357] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.058140 342 raft_consensus.cc:383] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.058249 342 raft_consensus.cc:738] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.058529 342 consensus_queue.cc:260] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.059118 342 ts_tablet_manager.cc:1428] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:42.059195 1984 tablet.cc:1722] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:42.059502 342 tablet_bootstrap.cc:492] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.061121 342 tablet_bootstrap.cc:654] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.061802 32054 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:42.061996 32054 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:42.062454 32054 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:42.062664 32054 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:42.062933 32054 catalog_manager.cc:5582] T 7874a1d5cb9a464294ca32b2767efbf3 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:42.063371 32054 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:42.063951 32054 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (table default.Table_b764664d1bdc4441907690b6616f3570 [id=ad7210d2665f463c8591e79cf62d15dc]). Expected version 1 got 0
I20250623 08:04:42.064203 32054 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:42.064466 32054 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet f8a5e35c1974456b8079ab8907c2652e (table default.Table_d2473e1e63c84b229243a1c676266ca6 [id=c63ada5e02d74e049460990450ab4962]). Expected version 1 got 0
I20250623 08:04:42.064805 32054 catalog_manager.cc:5582] T a38eecf4a94d4536b98d4f6e037a0094 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:42.067001 952 ts_tablet_manager.cc:1918] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:42.068611 979 ts_tablet_manager.cc:1918] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:42.074828 952 log.cc:1199] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/2f62ec83c88e43438b963e740d2c7004
I20250623 08:04:42.075316 952 ts_tablet_manager.cc:1939] T 2f62ec83c88e43438b963e740d2c7004 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:42.076526 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 220) took 5933 ms. Trace:
I20250623 08:04:42.078140 952 rpcz_store.cc:276] 0623 08:04:36.143005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.143071 (+ 66us) service_pool.cc:225] Handling call
0623 08:04:42.041413 (+5898342us) ts_tablet_manager.cc:1145] Deleting tablet 2f62ec83c88e43438b963e740d2c7004
0623 08:04:42.041421 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.050276 (+ 8855us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.066897 (+ 16621us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.076504 (+ 9607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5898132}
I20250623 08:04:42.078049 342 tablet_bootstrap.cc:492] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.079017 342 ts_tablet_manager.cc:1397] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20250623 08:04:42.079779 342 raft_consensus.cc:357] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.080018 342 raft_consensus.cc:383] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.080114 342 raft_consensus.cc:738] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.080339 342 consensus_queue.cc:260] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.081301 342 ts_tablet_manager.cc:1428] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:42.081593 342 tablet_bootstrap.cc:492] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:41.987315 1682 raft_consensus.cc:491] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.083810 1682 raft_consensus.cc:513] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.084142 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 222) took 5855 ms. Trace:
I20250623 08:04:42.084237 952 rpcz_store.cc:276] 0623 08:04:36.228497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.228549 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:42.083979 (+5855430us) ts_tablet_manager.cc:1145] Deleting tablet 16331ba6a34942a5a1b4ff4ac7bac55e
0623 08:04:42.083989 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.084123 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5855253}
I20250623 08:04:42.084412 1682 leader_election.cc:290] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:41.987965 339 tablet_bootstrap.cc:492] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:42.084645 339 ts_tablet_manager.cc:1397] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.143s user 0.000s sys 0.003s
I20250623 08:04:42.085289 339 raft_consensus.cc:357] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.085448 339 raft_consensus.cc:383] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.085498 339 raft_consensus.cc:738] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.085662 339 consensus_queue.cc:260] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.086114 979 log.cc:1199] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/6926c49b787941d98c647543bd3b9ed5
I20250623 08:04:42.086158 339 ts_tablet_manager.cc:1428] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250623 08:04:42.086447 339 tablet_bootstrap.cc:492] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:42.086493 979 ts_tablet_manager.cc:1939] T 6926c49b787941d98c647543bd3b9ed5 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:42.087035 342 tablet_bootstrap.cc:654] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.088068 339 tablet_bootstrap.cc:654] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:41.990106 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9f4829e103b460eb582beff3b966bb4" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.089250 32376 raft_consensus.cc:2466] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:41.990166 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "771ecfc41e2b4007a4d213cec5b128c6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.089582 32380 raft_consensus.cc:2466] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:41.990213 32358 tablet_service.cc:1515] Processing DeleteTablet for tablet ba5913811e9343389c9ea0ec8b8c05ef with delete_type TABLET_DATA_DELETED (replaced by tablet 2ec871b2878a4b3f929b3774746d7750) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990231 32349 tablet_service.cc:1515] Processing DeleteTablet for tablet 48da1e4759bb4fea87a45ad8a428b6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990252 32351 tablet_service.cc:1515] Processing DeleteTablet for tablet adc3cf4355c24b7395fcb22af8520346 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990314 32352 tablet_service.cc:1515] Processing DeleteTablet for tablet db81250d1e164a7d94affd764fc0ff0c with delete_type TABLET_DATA_DELETED (replaced by tablet 89a24019b797467e857b744d701fb1ed) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990329 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 789d1c20aa72480ab71123c799bbde4c with delete_type TABLET_DATA_DELETED (replaced by tablet 0cca63991d5b4ef5bd8a9181d127ebe3) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990352 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet 79731106915d4cebb9dcc3c10386ffa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:26 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990574 32342 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 077bb090b3ea4d029463304d7a6feee5) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990615 32348 tablet_service.cc:1515] Processing DeleteTablet for tablet 70f9d21b8c144f9cada9bdbe0fbb1538 with delete_type TABLET_DATA_DELETED (replaced by tablet db81250d1e164a7d94affd764fc0ff0c) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990653 32346 tablet_service.cc:1515] Processing DeleteTablet for tablet d0b2af698b6e4ed0ae66a9bb32be7b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:30 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990852 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet d19d468ce13d4108a9badc36fc503306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.990901 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 52d0a356c74f4daea9093c998b4bf73e with delete_type TABLET_DATA_DELETED (replaced by tablet bb06a340f82047ac8ec3516edb08f4b6) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.991106 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet 6926c49b787941d98c647543bd3b9ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:31 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:42.058797 1843 raft_consensus.cc:491] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.091140 1843 raft_consensus.cc:513] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.091791 1843 leader_election.cc:290] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.094844 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 223) took 5866 ms. Trace:
I20250623 08:04:42.094992 952 rpcz_store.cc:276] 0623 08:04:36.229477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.229531 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:42.084498 (+5854967us) ts_tablet_manager.cc:1145] Deleting tablet e22f9bae5f7248af99cec06ce6dbbd41
0623 08:04:42.084508 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.094825 (+ 10317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5854785}
I20250623 08:04:42.095355 952 tablet_replica.cc:331] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:42.095510 32043 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 6926c49b787941d98c647543bd3b9ed5 (table default.Table_168a55a099724943a6e452febfeae7b3 [id=ecddff52c4604b1d99a6f1cd6fa37641]) successfully deleted
I20250623 08:04:42.095582 952 raft_consensus.cc:2241] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:42.095800 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet ecb422550efd40cf82bc92fca5c25eda because the tablet was not found. No further retry: Not found: Tablet not found: ecb422550efd40cf82bc92fca5c25eda
I20250623 08:04:42.095856 952 raft_consensus.cc:2270] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:42.095976 32043 catalog_manager.cc:4895] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 025365d2fa0a43dd983d395a05cc2af1 because the tablet was not found. No further retry: Not found: Tablet not found: 025365d2fa0a43dd983d395a05cc2af1
W20250623 08:04:42.096170 32043 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 74c1cdc83ffd4a09a163479ac58fd405 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74c1cdc83ffd4a09a163479ac58fd405 already in progress: creating tablet
I20250623 08:04:42.075646 343 tablet_bootstrap.cc:492] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.096987 343 ts_tablet_manager.cc:1397] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.152s user 0.001s sys 0.005s
I20250623 08:04:42.097134 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45293db3c40b4e4194445f12e7475e66" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.096745 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.097536 32642 raft_consensus.cc:2391] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:42.097906 343 raft_consensus.cc:357] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.097980 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45293db3c40b4e4194445f12e7475e66" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.098150 343 raft_consensus.cc:383] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.098191 32514 raft_consensus.cc:2391] T 45293db3c40b4e4194445f12e7475e66 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:42.098219 343 raft_consensus.cc:738] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.098635 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b84b8b8713774ab4896842feb9fb075e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.098695 32314 leader_election.cc:304] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:42.098553 1986 tablet.cc:1722] T 9fd478c007914d2cb09d929b50322a64 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:42.099079 1682 raft_consensus.cc:3058] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.099671 952 ts_tablet_manager.cc:1905] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.101696 1682 raft_consensus.cc:2747] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:42.104445 32314 leader_election.cc:343] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.106433 32314 leader_election.cc:304] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:42.106768 343 consensus_queue.cc:260] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.107299 1682 raft_consensus.cc:3058] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.109059 1843 raft_consensus.cc:491] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.109193 1843 raft_consensus.cc:513] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.109818 1843 leader_election.cc:290] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.110105 342 tablet_bootstrap.cc:492] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.110236 342 ts_tablet_manager.cc:1397] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250623 08:04:42.110975 342 raft_consensus.cc:357] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.111135 342 raft_consensus.cc:383] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.111193 342 raft_consensus.cc:738] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.111354 1682 raft_consensus.cc:2747] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:42.111361 342 consensus_queue.cc:260] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.112705 343 ts_tablet_manager.cc:1428] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250623 08:04:41.990063 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 886fc0fa39c54bddb25ac01f68d95805) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:41.991014 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ed4cb252b1452087f8ee53ec78d985 with delete_type TABLET_DATA_DELETED (replaced by tablet 306c4021b5d94a4996de07c222a8bfd8) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:42.027422 1982 tablet.cc:1722] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:42.077577 1987 tablet.cc:1722] T 9fd478c007914d2cb09d929b50322a64 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250623 08:04:42.113888 32314 leader_election.cc:343] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: ecb422550efd40cf82bc92fca5c25eda
I20250623 08:04:42.111842 342 ts_tablet_manager.cc:1428] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:42.114956 342 tablet_bootstrap.cc:492] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:42.115917 32314 leader_election.cc:343] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: ecb422550efd40cf82bc92fca5c25eda
I20250623 08:04:42.116014 32314 leader_election.cc:304] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:42.116686 342 tablet_bootstrap.cc:654] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.116902 1682 raft_consensus.cc:2747] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:42.120695 1902 raft_consensus.cc:491] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:41.966727 981 ts_tablet_manager.cc:1918] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:42.120867 981 log.cc:1199] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/162308da898d4dde8ab5b1ab01d0f393
I20250623 08:04:42.120898 952 ts_tablet_manager.cc:1918] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:42.121001 1902 raft_consensus.cc:513] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.121177 981 ts_tablet_manager.cc:1939] T 162308da898d4dde8ab5b1ab01d0f393 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:42.122154 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3740dd120745493688eeb4c1ac311dcb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:42.122840 32589 leader_election.cc:343] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.123051 1682 raft_consensus.cc:491] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.123155 1682 raft_consensus.cc:513] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.123539 1902 leader_election.cc:290] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.123656 1682 leader_election.cc:290] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.124259 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f83f68ebda7e4826bbb17278cd4ac5a8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.124425 32647 raft_consensus.cc:2466] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:42.126075 32314 leader_election.cc:304] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:42.126322 1682 raft_consensus.cc:2802] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.126406 1682 raft_consensus.cc:491] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.126473 1682 raft_consensus.cc:3058] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.129166 1682 raft_consensus.cc:513] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.129801 1682 leader_election.cc:290] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.131556 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 265) took 5320 ms. Trace:
I20250623 08:04:42.131644 981 rpcz_store.cc:276] 0623 08:04:36.810833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.811044 (+ 211us) service_pool.cc:225] Handling call
0623 08:04:41.959589 (+5148545us) ts_tablet_manager.cc:1145] Deleting tablet 162308da898d4dde8ab5b1ab01d0f393
0623 08:04:41.959600 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:41.963801 (+ 4201us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:41.966634 (+ 2833us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.131539 (+164905us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5132614}
I20250623 08:04:42.131950 981 tablet_replica.cc:331] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:42.132145 981 raft_consensus.cc:2241] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:42.132320 981 raft_consensus.cc:2270] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.132853 981 ts_tablet_manager.cc:1905] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.134150 952 log.cc:1199] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b84b8b8713774ab4896842feb9fb075e
I20250623 08:04:42.134289 32631 tablet_service.cc:1468] Processing CreateTablet for tablet de8130ee0d4b4603834407e52fe3f9e1 (DEFAULT_TABLE table=default.Table_fb578541a1094c0f90ba5281797a457d [id=0ed6580f5bc04a48830d33c64804a1ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.121121 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:42.124272 1957 consensus_queue.cc:1035] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:42.126668 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f83f68ebda7e4826bbb17278cd4ac5a8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.134765 32627 tablet_service.cc:1468] Processing CreateTablet for tablet 8da872257b26441c8456446426affe7a (DEFAULT_TABLE table=default.Table_b91721eeaabb4377b4d62483ad065bee [id=8544c855ea7e4fed887765c7730c3ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.134851 32519 raft_consensus.cc:2466] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:42.134554 952 ts_tablet_manager.cc:1939] T b84b8b8713774ab4896842feb9fb075e P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:42.137775 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 224) took 5907 ms. Trace:
I20250623 08:04:42.137902 952 rpcz_store.cc:276] 0623 08:04:36.230004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.230055 (+ 51us) service_pool.cc:225] Handling call
0623 08:04:42.095312 (+5865257us) ts_tablet_manager.cc:1145] Deleting tablet b84b8b8713774ab4896842feb9fb075e
0623 08:04:42.095321 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.117664 (+ 22343us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.120816 (+ 3152us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.137760 (+ 16944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5865114}
I20250623 08:04:42.138337 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 226) took 5840 ms. Trace:
I20250623 08:04:42.138459 952 rpcz_store.cc:276] 0623 08:04:36.297552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.297624 (+ 72us) service_pool.cc:225] Handling call
0623 08:04:42.138227 (+5840603us) ts_tablet_manager.cc:1145] Deleting tablet 9bfa787f828b411ab68d81bb896f7fb6
0623 08:04:42.138236 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.138323 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5840365}
I20250623 08:04:42.139494 342 tablet_bootstrap.cc:492] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.139678 342 ts_tablet_manager.cc:1397] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250623 08:04:42.139925 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 2f62ec83c88e43438b963e740d2c7004 (table default.Table_c8a1668e7bb64df089cfbfbf4aeb90ac [id=65c4e79e095b41b2b2a42e9858485423]) successfully deleted
W20250623 08:04:42.140210 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 16331ba6a34942a5a1b4ff4ac7bac55e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16331ba6a34942a5a1b4ff4ac7bac55e already in progress: creating tablet
W20250623 08:04:42.140527 32043 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet e22f9bae5f7248af99cec06ce6dbbd41 because the tablet was not found. No further retry: Not found: Tablet not found: e22f9bae5f7248af99cec06ce6dbbd41
I20250623 08:04:42.140681 342 raft_consensus.cc:357] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.140872 342 raft_consensus.cc:383] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.140967 342 raft_consensus.cc:738] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.141216 342 consensus_queue.cc:260] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.141760 342 ts_tablet_manager.cc:1428] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:42.141980 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 235) took 5677 ms. Trace:
I20250623 08:04:42.142352 952 rpcz_store.cc:276] 0623 08:04:36.464984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.468892 (+ 3908us) service_pool.cc:225] Handling call
0623 08:04:42.138733 (+5669841us) ts_tablet_manager.cc:1145] Deleting tablet f5f58f55070344ff98a3808f70e0c755
0623 08:04:42.138742 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.141964 (+ 3222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5656651}
I20250623 08:04:42.142720 952 tablet_replica.cc:331] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:42.142295 342 tablet_bootstrap.cc:492] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.143213 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f83f68ebda7e4826bbb17278cd4ac5a8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:42.143195 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f83f68ebda7e4826bbb17278cd4ac5a8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:42.143352 32519 raft_consensus.cc:3058] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.143359 32647 raft_consensus.cc:3058] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.144758 342 tablet_bootstrap.cc:654] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.145685 952 raft_consensus.cc:2241] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:42.146090 32647 raft_consensus.cc:2466] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:42.146121 32519 raft_consensus.cc:2466] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:42.146708 32359 tablet_service.cc:1468] Processing CreateTablet for tablet de8130ee0d4b4603834407e52fe3f9e1 (DEFAULT_TABLE table=default.Table_fb578541a1094c0f90ba5281797a457d [id=0ed6580f5bc04a48830d33c64804a1ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.147161 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8130ee0d4b4603834407e52fe3f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.147183 32043 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 162308da898d4dde8ab5b1ab01d0f393 (table default.Table_c30b48097bd643b18b2a744839780b9e [id=7e59ad2235c1418d97f2b73ff2a48300]) successfully deleted
I20250623 08:04:42.151355 32622 tablet_service.cc:1468] Processing CreateTablet for tablet d47c48aeed064015868f50c2c0dee132 (DEFAULT_TABLE table=default.Table_1802c9f70d0f45868e11e3390504fade [id=15591f6f911f4a91a1dbbef45354fe1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.151732 32618 tablet_service.cc:1468] Processing CreateTablet for tablet c363a0602836433a8fd41fde529e96f3 (DEFAULT_TABLE table=default.Table_88d6a5158a8d4ae192cf71e8eaffff54 [id=e218dda26a86485a9983a0bc74c4c89f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.155558 1953 raft_consensus.cc:491] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.155670 952 raft_consensus.cc:2270] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.155689 1953 raft_consensus.cc:513] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.156210 952 ts_tablet_manager.cc:1905] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.156380 1953 leader_election.cc:290] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.157394 32636 tablet_service.cc:1468] Processing CreateTablet for tablet f1113750ec7a4dfe8a424afe0d41c33b (DEFAULT_TABLE table=default.Table_eb0f697365cc4f2b8bbeb5b3de47f1dc [id=f15f4f7ed5054a3ca292217b18a14a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.158300 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 15d9b4e331414d3eace2b4098b13731e (DEFAULT_TABLE table=default.Table_070eb9d8f3af4ed9bb823cf579ade9c8 [id=8f90ed2f56464b9f8dc494452e6c6aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.163085 32453 leader_election.cc:304] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:42.163376 1953 raft_consensus.cc:2802] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.163470 1953 raft_consensus.cc:491] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.163524 1953 raft_consensus.cc:3058] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.163846 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 8da872257b26441c8456446426affe7a (DEFAULT_TABLE table=default.Table_b91721eeaabb4377b4d62483ad065bee [id=8544c855ea7e4fed887765c7730c3ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.164183 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8da872257b26441c8456446426affe7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.166260 1953 raft_consensus.cc:513] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.169996 1953 leader_election.cc:290] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.170909 32353 tablet_service.cc:1468] Processing CreateTablet for tablet c363a0602836433a8fd41fde529e96f3 (DEFAULT_TABLE table=default.Table_88d6a5158a8d4ae192cf71e8eaffff54 [id=e218dda26a86485a9983a0bc74c4c89f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.171262 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c363a0602836433a8fd41fde529e96f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.175338 32634 tablet_service.cc:1468] Processing CreateTablet for tablet a79238b6f6e74ae89c52eefb95190264 (DEFAULT_TABLE table=default.Table_5d6ec4f6805b4e0883b8d0419625d033 [id=545b478879aa4f4cb35393e3065d3ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.177019 342 tablet_bootstrap.cc:492] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.177163 342 ts_tablet_manager.cc:1397] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.002s
I20250623 08:04:42.177560 32626 tablet_service.cc:1468] Processing CreateTablet for tablet c50655c06cd94c67a49513c39728eaa9 (DEFAULT_TABLE table=default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.178217 32630 tablet_service.cc:1468] Processing CreateTablet for tablet d7caf974de5049d3b6db2754970ed2e3 (DEFAULT_TABLE table=default.Table_0e43b0bc6b7b47b1b59882856aaa9736 [id=30949c1b60bc4300852bcaf533ceed4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.178869 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 257227ae140446d78117b6150dbb3962 (DEFAULT_TABLE table=default.Table_bd4ee5f643fe4d9bab6b2f11ef0869cf [id=28106e3ea0bd442e85bbbe3e26f8f0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.180927 32353 tablet_service.cc:1468] Processing CreateTablet for tablet f1113750ec7a4dfe8a424afe0d41c33b (DEFAULT_TABLE table=default.Table_eb0f697365cc4f2b8bbeb5b3de47f1dc [id=f15f4f7ed5054a3ca292217b18a14a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.181332 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1113750ec7a4dfe8a424afe0d41c33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.187232 1954 raft_consensus.cc:491] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.187362 1954 raft_consensus.cc:513] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.187529 342 raft_consensus.cc:357] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.187650 342 raft_consensus.cc:383] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.187716 342 raft_consensus.cc:738] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.187898 342 consensus_queue.cc:260] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.187988 1954 leader_election.cc:290] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.188424 342 ts_tablet_manager.cc:1428] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250623 08:04:42.188488 1984 tablet.cc:1722] T 2b405f87bb1e4d0686a1f627835d901f P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:42.188697 342 tablet_bootstrap.cc:492] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.188654 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3740dd120745493688eeb4c1ac311dcb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.188807 32514 raft_consensus.cc:2466] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:42.190352 342 tablet_bootstrap.cc:654] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.195183 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 15d9b4e331414d3eace2b4098b13731e (DEFAULT_TABLE table=default.Table_070eb9d8f3af4ed9bb823cf579ade9c8 [id=8f90ed2f56464b9f8dc494452e6c6aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.195600 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d9b4e331414d3eace2b4098b13731e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.199368 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:42.201264 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]). Expected version 1 got 0
I20250623 08:04:42.201845 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:42.203281 32357 tablet_service.cc:1468] Processing CreateTablet for tablet c50655c06cd94c67a49513c39728eaa9 (DEFAULT_TABLE table=default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.203652 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50655c06cd94c67a49513c39728eaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.207358 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:42.208477 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f8a5e35c1974456b8079ab8907c2652e (table default.Table_d2473e1e63c84b229243a1c676266ca6 [id=c63ada5e02d74e049460990450ab4962]). Expected version 1 got 0
I20250623 08:04:42.208922 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:42.209218 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:42.209493 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:42.209707 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:42.210326 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (table default.Table_b764664d1bdc4441907690b6616f3570 [id=ad7210d2665f463c8591e79cf62d15dc]). Expected version 1 got 0
I20250623 08:04:42.210578 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:42.211215 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 257227ae140446d78117b6150dbb3962 (DEFAULT_TABLE table=default.Table_bd4ee5f643fe4d9bab6b2f11ef0869cf [id=28106e3ea0bd442e85bbbe3e26f8f0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.211712 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 257227ae140446d78117b6150dbb3962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.212738 342 tablet_bootstrap.cc:492] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.212880 342 ts_tablet_manager.cc:1397] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20250623 08:04:42.213466 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:42.213655 342 raft_consensus.cc:357] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.213843 342 raft_consensus.cc:383] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.213904 342 raft_consensus.cc:738] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.214162 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]). Expected version 1 got 0
I20250623 08:04:42.214134 342 consensus_queue.cc:260] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.214622 342 ts_tablet_manager.cc:1428] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:42.215171 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:42.215494 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:42.215839 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:42.216025 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:42.216226 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:42.216461 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:42.216653 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:42.216907 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:42.217384 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:42.217816 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:42.218173 32354 tablet_service.cc:1468] Processing CreateTablet for tablet a79238b6f6e74ae89c52eefb95190264 (DEFAULT_TABLE table=default.Table_5d6ec4f6805b4e0883b8d0419625d033 [id=545b478879aa4f4cb35393e3065d3ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.218608 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a79238b6f6e74ae89c52eefb95190264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.218950 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:42.219573 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:42.219879 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:42.220201 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:42.220465 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:42.220814 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1c529e826fa84a919c018887620e4023 (table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]). Expected version 1 got 0
I20250623 08:04:42.221014 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:42.221392 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:42.222348 342 tablet_bootstrap.cc:492] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.222635 32357 tablet_service.cc:1468] Processing CreateTablet for tablet d7caf974de5049d3b6db2754970ed2e3 (DEFAULT_TABLE table=default.Table_0e43b0bc6b7b47b1b59882856aaa9736 [id=30949c1b60bc4300852bcaf533ceed4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.222963 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7caf974de5049d3b6db2754970ed2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.098047 979 tablet_replica.cc:331] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:42.224262 342 tablet_bootstrap.cc:654] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.227178 979 raft_consensus.cc:2241] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:42.229555 1682 raft_consensus.cc:491] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.229775 1682 raft_consensus.cc:513] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.222034 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1cee3de5953244f98428747bea912f77 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]). Expected version 1 got 0
I20250623 08:04:42.238664 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_217651efa5dc4571b2a47cd3196c9e57 [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 1 got 0
I20250623 08:04:42.229395 343 tablet_bootstrap.cc:492] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.229848 32631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8130ee0d4b4603834407e52fe3f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.240118 32634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a79238b6f6e74ae89c52eefb95190264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.240394 32622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d47c48aeed064015868f50c2c0dee132. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.243002 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 257227ae140446d78117b6150dbb3962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.244315 32627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8da872257b26441c8456446426affe7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.247408 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d9b4e331414d3eace2b4098b13731e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.248100 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:42.248508 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:42.249282 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:42.249611 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:42.249935 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
I20250623 08:04:42.250217 32057 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:42.250430 32055 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 1be8f8557b9a4ae2917c37b8d01cedf6
I20250623 08:04:42.251040 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1113750ec7a4dfe8a424afe0d41c33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.251214 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_4d2fc00363204f8485905af018f7911a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:42.251819 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d2fc00363204f8485905af018f7911a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:42.239862 342 tablet_bootstrap.cc:492] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.252920 342 ts_tablet_manager.cc:1397] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250623 08:04:42.253643 342 raft_consensus.cc:357] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.253793 342 raft_consensus.cc:383] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.253834 342 raft_consensus.cc:738] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.254001 342 consensus_queue.cc:260] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.254483 342 ts_tablet_manager.cc:1428] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:42.254608 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50655c06cd94c67a49513c39728eaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.254791 342 tablet_bootstrap.cc:492] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.255146 1954 raft_consensus.cc:491] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.255251 1954 raft_consensus.cc:513] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.255813 1954 leader_election.cc:290] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.256449 342 tablet_bootstrap.cc:654] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.258014 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c363a0602836433a8fd41fde529e96f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.261848 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7caf974de5049d3b6db2754970ed2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.265770 1954 raft_consensus.cc:491] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.265909 1954 raft_consensus.cc:513] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.266326 1902 raft_consensus.cc:491] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.266520 1954 leader_election.cc:290] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.266517 1902 raft_consensus.cc:513] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.266790 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_87e0e2fcec324a53ae2cfd5b3156e071"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:42.267165 1902 leader_election.cc:290] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:42.267468 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87e0e2fcec324a53ae2cfd5b3156e071 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:42.283555 32055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_1ef88ab1f1cd49139106aed3732b50a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:42.284873 32055 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ef88ab1f1cd49139106aed3732b50a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:42.285091 32627 tablet_service.cc:1515] Processing DeleteTablet for tablet 1be8f8557b9a4ae2917c37b8d01cedf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:42 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:42.295691 32055 master_service.cc:948] ReplaceTablet: received request to replace tablet 9fd478c007914d2cb09d929b50322a64 from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:42.296320 32051 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet 3740dd120745493688eeb4c1ac311dcb
I20250623 08:04:42.296288 32058 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet a8de2b1105564fdab386442b1de4cedb
I20250623 08:04:42.296891 32058 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 395) took 2351 ms. Trace:
I20250623 08:04:42.297291 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_47c06fdd0628467d8a0727b9adeb890d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:42.297003 32058 rpcz_store.cc:276] 0623 08:04:39.945716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.221765 (+1276049us) service_pool.cc:225] Handling call
0623 08:04:41.303057 (+ 81292us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:41.303243 (+ 186us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:41.303320 (+ 77us) catalog_manager.cc:2799] Locking tablets
0623 08:04:41.303538 (+ 218us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:41.499648 (+196110us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:41.499744 (+ 96us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:42.296222 (+796478us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:42.296828 (+ 606us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.321039 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.321225 (+ 186us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.321233 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:41.321237 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:41.321439 (+ 202us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.321453 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.321460 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:41.321571 (+ 111us) tablet.cc:699] Row locks acquired
0623 08:04:41.321573 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:41.321604 (+ 31us) write_op.cc:270] Start()
0623 08:04:41.321635 (+ 31us) write_op.cc:276] Timestamp: P: 1750665881321546 usec, L: 0
0623 08:04:41.321639 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.464693 (+143054us) log.cc:844] Serialized 756 byte log entry
0623 08:04:41.464907 (+ 214us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x616000031054. stack: 00007f19bd08ed56 00007f19bd08eb10 00007f19bc08529d 0000000000970a84 0000000000970a4d 000000000097041a 00007f19c8ac25e2 00007f19c9e69255 00007f19c9e67d79 00007f19c9e701a0 00007f19c9e6ffad 0000000000a751bc 00007f19bd0d43ff 00007f19bd0d76cb 00007f19bd0d7548 0000000000a751bc
0623 08:04:41.499040 (+ 34133us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.499133 (+ 93us) write_op.cc:301] APPLY: starting
0623 08:04:41.499149 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.499188 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.499191 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.499287 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.499320 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:41.499328 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:41.499405 (+ 77us) log.cc:844] Serialized 29 byte log entry
0623 08:04:41.499493 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.499517 (+ 24us) write_op.cc:454] Released schema lock
0623 08:04:41.499525 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":40,"apply.run_cpu_time_us":453,"apply.run_wall_time_us":456,"num_ops":2,"prepare.queue_time_us":16635,"prepare.run_cpu_time_us":2699,"prepare.run_wall_time_us":143947,"raft.queue_time_us":8179,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173,"replication_time_us":177372,"spinlock_wait_cycles":328420224}]]}
I20250623 08:04:42.297853 32058 master_service.cc:948] ReplaceTablet: received request to replace tablet a2a07188a4d340bc9caf010d5202e34d from {username='slave'} at 127.0.0.1:36116
W20250623 08:04:42.297996 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47c06fdd0628467d8a0727b9adeb890d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:42.301833 343 tablet_bootstrap.cc:654] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.303936 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 3740dd120745493688eeb4c1ac311dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:42.306111 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet a8de2b1105564fdab386442b1de4cedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:42.307916 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_100f074733ca4066983e68257aea0b11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:42.309347 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_100f074733ca4066983e68257aea0b11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:42.313395 32059 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca] alter complete (version 1)
I20250623 08:04:42.319175 32051 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_992eb98f4db0454083b215b3ac3d38b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:42.319697 32051 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_992eb98f4db0454083b215b3ac3d38b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:42.320861 32043 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b84b8b8713774ab4896842feb9fb075e (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]) successfully deleted
W20250623 08:04:42.321257 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 9bfa787f828b411ab68d81bb896f7fb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bfa787f828b411ab68d81bb896f7fb6 already in progress: creating tablet
W20250623 08:04:42.321877 32043 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet f5f58f55070344ff98a3808f70e0c755 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5f58f55070344ff98a3808f70e0c755 already in progress: creating tablet
W20250623 08:04:42.323798 32056 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7f890ee0747b4df0b82bc3b396191124 consensus state has no leader
I20250623 08:04:42.324366 32056 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_cace1e64be664c9c81b588e80d704aaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:42.324936 979 raft_consensus.cc:2270] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:42.325048 32056 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cace1e64be664c9c81b588e80d704aaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:42.325675 979 ts_tablet_manager.cc:1905] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.327014 32051 master_service.cc:948] ReplaceTablet: received request to replace tablet b84b8b8713774ab4896842feb9fb075e from {username='slave'} at 127.0.0.1:36116
I20250623 08:04:42.327965 32054 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517] alter complete (version 5)
I20250623 08:04:42.329912 32052 catalog_manager.cc:5869] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Sending DeleteTablet for 3 replicas of tablet a9ea74d20c5f4eefbcb8508ed1774026
W20250623 08:04:42.330099 32050 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7b9e843ff1344a1aaacf038f15f64cd consensus state has no leader
I20250623 08:04:42.330506 32052 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36116 (request call id 404) took 1491 ms. Trace:
I20250623 08:04:42.330578 32050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_1631659a168049fba9411207edd8b56f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:42.330621 32052 rpcz_store.cc:276] 0623 08:04:40.838999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.469849 (+630850us) service_pool.cc:225] Handling call
0623 08:04:41.470265 (+ 416us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:41.470407 (+ 142us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:41.470474 (+ 67us) catalog_manager.cc:2799] Locking tablets
0623 08:04:41.470636 (+ 162us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:41.564672 (+ 94036us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:41.564769 (+ 97us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:42.329842 (+765073us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:42.330465 (+ 623us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:41.495093 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:41.495194 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:41.495201 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:41.495205 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:41.495341 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:41.495350 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:41.495354 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:41.495454 (+ 100us) tablet.cc:699] Row locks acquired
0623 08:04:41.495469 (+ 15us) write_op.cc:260] PREPARE: finished
0623 08:04:41.495499 (+ 30us) write_op.cc:270] Start()
0623 08:04:41.495526 (+ 27us) write_op.cc:276] Timestamp: P: 1750665881495461 usec, L: 0
0623 08:04:41.495530 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:41.495674 (+ 144us) log.cc:844] Serialized 790 byte log entry
0623 08:04:41.563940 (+ 68266us) op_driver.cc:464] REPLICATION: finished
0623 08:04:41.564127 (+ 187us) write_op.cc:301] APPLY: starting
0623 08:04:41.564140 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:41.564174 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:41.564205 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:41.564274 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:41.564312 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:41.564324 (+ 12us) write_op.cc:312] APPLY: finished
0623 08:04:41.564384 (+ 60us) log.cc:844] Serialized 29 byte log entry
0623 08:04:41.564439 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:41.564467 (+ 28us) write_op.cc:454] Released schema lock
0623 08:04:41.564473 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":142,"apply.run_cpu_time_us":417,"apply.run_wall_time_us":417,"num_ops":2,"prepare.queue_time_us":23808,"prepare.run_cpu_time_us":662,"prepare.run_wall_time_us":3668,"replication_time_us":68384}]]}
W20250623 08:04:42.331249 32050 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1631659a168049fba9411207edd8b56f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:42.331385 32052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36116:
name: "default.Table_14e9f0f5ff5d42d9bdecf7f91482d8b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:42.331892 32052 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14e9f0f5ff5d42d9bdecf7f91482d8b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:42.333109 1682 leader_election.cc:290] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.336557 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd7f3d2e67e84bfcbd45a21658f69e7b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.336743 32642 raft_consensus.cc:2466] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.337129 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e898ede64e641519c8ee3ea1853a4b0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.337391 32647 raft_consensus.cc:2466] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.343092 32314 leader_election.cc:304] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:42.343619 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet a9ea74d20c5f4eefbcb8508ed1774026 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:36612
I20250623 08:04:42.344380 32492 tablet_service.cc:1468] Processing CreateTablet for tablet de8130ee0d4b4603834407e52fe3f9e1 (DEFAULT_TABLE table=default.Table_fb578541a1094c0f90ba5281797a457d [id=0ed6580f5bc04a48830d33c64804a1ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.344390 32484 tablet_service.cc:1468] Processing CreateTablet for tablet 8da872257b26441c8456446426affe7a (DEFAULT_TABLE table=default.Table_b91721eeaabb4377b4d62483ad065bee [id=8544c855ea7e4fed887765c7730c3ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.344832 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8da872257b26441c8456446426affe7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.347030 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8130ee0d4b4603834407e52fe3f9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.354669 1954 raft_consensus.cc:491] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.354838 1954 raft_consensus.cc:513] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.355538 1954 leader_election.cc:290] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.356665 32051 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_217651efa5dc4571b2a47cd3196c9e57" } new_table_name: "default.Table_8f362905029746c19e0e7ce1433c594d" modify_external_catalogs: true
I20250623 08:04:42.359405 32054 catalog_manager.cc:6127] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5] alter complete (version 1)
I20250623 08:04:42.366155 32020 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskJxinPB/build/asan/bin/kudu pid 32028
I20250623 08:04:42.367252 32089 generic_service.cc:196] Checking for leaks (request via RPC)
I20250623 08:04:42.371639 32052 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36116:
table { table_name: "default.Table_b68f23253200457683bb5cedcc56041f" } new_table_name: "default.Table_682a42dbf21d43d49ad9d85b958b121c" modify_external_catalogs: true
I20250623 08:04:42.377697 343 tablet_bootstrap.cc:492] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.377923 343 ts_tablet_manager.cc:1397] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.149s user 0.002s sys 0.001s
I20250623 08:04:42.378719 343 raft_consensus.cc:357] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.379196 343 raft_consensus.cc:383] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.379297 343 raft_consensus.cc:738] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.379573 343 consensus_queue.cc:260] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.382867 343 ts_tablet_manager.cc:1428] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250623 08:04:42.383180 343 tablet_bootstrap.cc:492] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.384811 343 tablet_bootstrap.cc:654] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.391043 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6fdea59f71e4a9d9fca8ede4b9439d2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.391587 952 ts_tablet_manager.cc:1918] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:42.391800 952 log.cc:1199] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/48da1e4759bb4fea87a45ad8a428b6e7
W20250623 08:04:42.393045 32589 leader_election.cc:343] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.393702 32489 tablet_service.cc:1468] Processing CreateTablet for tablet c363a0602836433a8fd41fde529e96f3 (DEFAULT_TABLE table=default.Table_88d6a5158a8d4ae192cf71e8eaffff54 [id=e218dda26a86485a9983a0bc74c4c89f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.393774 32494 tablet_service.cc:1468] Processing CreateTablet for tablet 15d9b4e331414d3eace2b4098b13731e (DEFAULT_TABLE table=default.Table_070eb9d8f3af4ed9bb823cf579ade9c8 [id=8f90ed2f56464b9f8dc494452e6c6aed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.394179 32500 tablet_service.cc:1468] Processing CreateTablet for tablet f1113750ec7a4dfe8a424afe0d41c33b (DEFAULT_TABLE table=default.Table_eb0f697365cc4f2b8bbeb5b3de47f1dc [id=f15f4f7ed5054a3ca292217b18a14a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.394523 952 ts_tablet_manager.cc:1939] T 48da1e4759bb4fea87a45ad8a428b6e7 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:42.395452 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c363a0602836433a8fd41fde529e96f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.395743 32483 tablet_service.cc:1468] Processing CreateTablet for tablet d47c48aeed064015868f50c2c0dee132 (DEFAULT_TABLE table=default.Table_1802c9f70d0f45868e11e3390504fade [id=15591f6f911f4a91a1dbbef45354fe1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.396092 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d9b4e331414d3eace2b4098b13731e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.399369 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d47c48aeed064015868f50c2c0dee132. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.401367 32485 tablet_service.cc:1468] Processing CreateTablet for tablet a79238b6f6e74ae89c52eefb95190264 (DEFAULT_TABLE table=default.Table_5d6ec4f6805b4e0883b8d0419625d033 [id=545b478879aa4f4cb35393e3065d3ff7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.401644 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a79238b6f6e74ae89c52eefb95190264. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.402456 32486 tablet_service.cc:1468] Processing CreateTablet for tablet c50655c06cd94c67a49513c39728eaa9 (DEFAULT_TABLE table=default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.402738 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c50655c06cd94c67a49513c39728eaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.404870 339 tablet_bootstrap.cc:492] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:42.404989 339 ts_tablet_manager.cc:1397] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.319s user 0.003s sys 0.003s
I20250623 08:04:42.405471 32499 tablet_service.cc:1468] Processing CreateTablet for tablet d7caf974de5049d3b6db2754970ed2e3 (DEFAULT_TABLE table=default.Table_0e43b0bc6b7b47b1b59882856aaa9736 [id=30949c1b60bc4300852bcaf533ceed4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.405787 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7caf974de5049d3b6db2754970ed2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.407285 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1113750ec7a4dfe8a424afe0d41c33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.409699 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 240) took 5920 ms. Trace:
I20250623 08:04:42.409847 952 rpcz_store.cc:276] 0623 08:04:36.489105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.489596 (+ 491us) service_pool.cc:225] Handling call
0623 08:04:42.142679 (+5653083us) ts_tablet_manager.cc:1145] Deleting tablet 48da1e4759bb4fea87a45ad8a428b6e7
0623 08:04:42.142688 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.160509 (+ 17821us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.391463 (+230954us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.409679 (+ 18216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9425,"tablet-delete.queue_time_us":5646921}
I20250623 08:04:42.410430 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 241) took 5921 ms. Trace:
I20250623 08:04:42.410566 952 rpcz_store.cc:276] 0623 08:04:36.489211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.489622 (+ 411us) service_pool.cc:225] Handling call
0623 08:04:42.410308 (+5920686us) ts_tablet_manager.cc:1145] Deleting tablet b61b02cf2904404f8c0ffef7d268fefc
0623 08:04:42.410319 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.410414 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5882939}
I20250623 08:04:42.410805 1910 raft_consensus.cc:2802] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.410989 1910 raft_consensus.cc:695] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:42.411206 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 242) took 5669 ms. Trace:
I20250623 08:04:42.411334 952 rpcz_store.cc:276] 0623 08:04:36.741700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.741756 (+ 56us) service_pool.cc:225] Handling call
0623 08:04:42.411029 (+5669273us) ts_tablet_manager.cc:1145] Deleting tablet b720fdea539f423ba2d2aa9f89b09ae9
0623 08:04:42.411041 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.411188 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5658937}
I20250623 08:04:42.411262 1910 consensus_queue.cc:237] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.411731 952 tablet_replica.cc:331] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: stopping tablet replica
W20250623 08:04:42.413651 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 70589.5us
I20250623 08:04:42.422298 343 tablet_bootstrap.cc:492] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.423048 343 ts_tablet_manager.cc:1397] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.013s
I20250623 08:04:42.423467 339 raft_consensus.cc:357] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.423694 339 raft_consensus.cc:383] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.423769 339 raft_consensus.cc:738] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.423995 339 consensus_queue.cc:260] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.424482 343 raft_consensus.cc:357] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.424612 339 ts_tablet_manager.cc:1428] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250623 08:04:42.425931 343 raft_consensus.cc:383] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.426029 1902 consensus_queue.cc:1035] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:42.426149 343 raft_consensus.cc:738] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.426604 343 consensus_queue.cc:260] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.426862 32376 raft_consensus.cc:3058] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.428812 339 tablet_bootstrap.cc:492] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:42.434082 343 ts_tablet_manager.cc:1428] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250623 08:04:42.434826 343 tablet_bootstrap.cc:492] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.436771 343 tablet_bootstrap.cc:654] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.454347 339 tablet_bootstrap.cc:654] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.464051 32359 tablet_service.cc:1468] Processing CreateTablet for tablet d47c48aeed064015868f50c2c0dee132 (DEFAULT_TABLE table=default.Table_1802c9f70d0f45868e11e3390504fade [id=15591f6f911f4a91a1dbbef45354fe1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.464510 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d47c48aeed064015868f50c2c0dee132. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.487915 342 tablet_bootstrap.cc:492] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.488128 342 ts_tablet_manager.cc:1397] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.233s user 0.042s sys 0.044s
I20250623 08:04:42.488936 342 raft_consensus.cc:357] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.489178 342 raft_consensus.cc:383] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.489265 342 raft_consensus.cc:738] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.489507 342 consensus_queue.cc:260] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.449009 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 257227ae140446d78117b6150dbb3962 (DEFAULT_TABLE table=default.Table_bd4ee5f643fe4d9bab6b2f11ef0869cf [id=28106e3ea0bd442e85bbbe3e26f8f0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:42.450850 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 3740dd120745493688eeb4c1ac311dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:42.450973 32484 tablet_service.cc:1515] Processing DeleteTablet for tablet 1be8f8557b9a4ae2917c37b8d01cedf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:42 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:42.490159 342 ts_tablet_manager.cc:1428] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250623 08:04:42.467480 32453 leader_election.cc:343] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 771ecfc41e2b4007a4d213cec5b128c6
I20250623 08:04:42.490509 32453 leader_election.cc:304] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.467626 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet a9ea74d20c5f4eefbcb8508ed1774026 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:42.474354 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet a8de2b1105564fdab386442b1de4cedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:55526
I20250623 08:04:42.491030 1953 raft_consensus.cc:2802] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.491106 1953 raft_consensus.cc:491] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.491176 1953 raft_consensus.cc:3058] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.491077 32453 leader_election.cc:304] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.491653 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e898ede64e641519c8ee3ea1853a4b0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.491727 1359 raft_consensus.cc:2802] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.491791 1359 raft_consensus.cc:491] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.491844 1359 raft_consensus.cc:3058] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.492141 32589 leader_election.cc:304] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:42.492506 1902 raft_consensus.cc:2802] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.492568 1902 raft_consensus.cc:491] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.492617 1902 raft_consensus.cc:3058] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.493767 1953 raft_consensus.cc:513] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.494313 1953 leader_election.cc:290] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.495275 1902 raft_consensus.cc:513] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.495904 1902 leader_election.cc:290] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.495383 1359 raft_consensus.cc:513] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.496645 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3740dd120745493688eeb4c1ac311dcb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.496896 1359 leader_election.cc:290] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:42.497282 32589 leader_election.cc:343] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.497365 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 257227ae140446d78117b6150dbb3962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:42.499943 32643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c5ecd4fa5574a1b90b6e82f89ee88f7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.500115 32643 raft_consensus.cc:2466] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.500614 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16bc969d09f74b0986070a504f9f670e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.500782 32649 raft_consensus.cc:2466] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.499445 1953 raft_consensus.cc:491] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.501223 1953 raft_consensus.cc:513] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.501312 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d1ec9d56ba40128325a50f204fe920" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.501466 32646 raft_consensus.cc:2466] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.501754 1953 leader_election.cc:290] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.506983 1902 consensus_queue.cc:1035] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:42.507426 342 tablet_bootstrap.cc:492] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.508807 952 raft_consensus.cc:2241] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:42.510159 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "771ecfc41e2b4007a4d213cec5b128c6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.510360 32376 raft_consensus.cc:3058] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.511554 979 ts_tablet_manager.cc:1918] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:42.509202 342 tablet_bootstrap.cc:654] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.511994 952 raft_consensus.cc:2270] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.512933 1627 raft_consensus.cc:491] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.513073 1627 raft_consensus.cc:513] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.513706 1627 leader_election.cc:290] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.514561 952 ts_tablet_manager.cc:1905] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.515125 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6fdea59f71e4a9d9fca8ede4b9439d2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.515653 32518 raft_consensus.cc:2466] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:42.519870 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd7f3d2e67e84bfcbd45a21658f69e7b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.521116 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3740dd120745493688eeb4c1ac311dcb" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:42.521240 32518 raft_consensus.cc:3058] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.523669 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16bc969d09f74b0986070a504f9f670e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.525172 32518 raft_consensus.cc:2466] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:42.534176 32376 raft_consensus.cc:2466] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.541803 32453 leader_election.cc:304] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.542115 1359 raft_consensus.cc:2802] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.542196 1359 raft_consensus.cc:491] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.542248 1359 raft_consensus.cc:3058] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.543622 32453 leader_election.cc:304] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.543838 1953 raft_consensus.cc:2802] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.543898 1953 raft_consensus.cc:491] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.543952 1953 raft_consensus.cc:3058] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.544211 342 tablet_bootstrap.cc:492] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.544318 342 ts_tablet_manager.cc:1397] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.004s
I20250623 08:04:42.544951 1359 raft_consensus.cc:513] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.545029 342 raft_consensus.cc:357] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.545171 342 raft_consensus.cc:383] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.525480 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d1ec9d56ba40128325a50f204fe920" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.545213 342 raft_consensus.cc:738] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.545300 32372 raft_consensus.cc:2466] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.545467 1359 leader_election.cc:290] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.545444 342 consensus_queue.cc:260] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.545972 342 ts_tablet_manager.cc:1428] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:42.546219 342 tablet_bootstrap.cc:492] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.546347 1953 raft_consensus.cc:513] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.546880 1953 leader_election.cc:290] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.546943 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c5ecd4fa5574a1b90b6e82f89ee88f7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.547581 32453 leader_election.cc:304] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.547784 1953 raft_consensus.cc:2802] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.547863 1953 raft_consensus.cc:491] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.547950 1953 raft_consensus.cc:3058] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.547996 342 tablet_bootstrap.cc:654] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.549328 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd7f3d2e67e84bfcbd45a21658f69e7b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.549863 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e898ede64e641519c8ee3ea1853a4b0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.550566 1953 raft_consensus.cc:513] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.551210 1953 leader_election.cc:290] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.551365 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1be8f8557b9a4ae2917c37b8d01cedf6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.552364 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e17e1ca404b462da43aa1db7dc3b556" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.553583 32354 tablet_service.cc:1515] Processing DeleteTablet for tablet a8de2b1105564fdab386442b1de4cedb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:42.553659 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet 3740dd120745493688eeb4c1ac311dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:42.554904 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet a9ea74d20c5f4eefbcb8508ed1774026 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:42.554996 342 tablet_bootstrap.cc:492] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.555110 342 ts_tablet_manager.cc:1397] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250623 08:04:42.555855 342 raft_consensus.cc:357] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.556006 342 raft_consensus.cc:383] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.556053 342 raft_consensus.cc:738] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.556151 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5af1dea3c6ca4f498ce6c039d72a14b7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.556205 342 consensus_queue.cc:260] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.556344 32518 raft_consensus.cc:2391] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:42.556651 342 ts_tablet_manager.cc:1428] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:42.525895 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd7f3d2e67e84bfcbd45a21658f69e7b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:42.557046 32647 raft_consensus.cc:3058] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.558843 342 tablet_bootstrap.cc:492] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.559139 32376 raft_consensus.cc:3058] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.559693 32647 raft_consensus.cc:2466] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.526535 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2e898ede64e641519c8ee3ea1853a4b0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:42.559938 32646 raft_consensus.cc:3058] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.560429 342 tablet_bootstrap.cc:654] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.562409 32646 raft_consensus.cc:2466] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.532544 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1be8f8557b9a4ae2917c37b8d01cedf6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.565009 32656 raft_consensus.cc:2466] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.535732 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e17e1ca404b462da43aa1db7dc3b556" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.565382 32651 raft_consensus.cc:2466] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.537545 32589 leader_election.cc:304] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:42.567080 1789 raft_consensus.cc:2802] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.567150 1789 raft_consensus.cc:491] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.567209 1789 raft_consensus.cc:3058] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.569445 981 ts_tablet_manager.cc:1918] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:42.569566 981 log.cc:1199] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/9a0bbafe1ea34ab5ae1b3de25b50d4a3
I20250623 08:04:42.569566 1789 raft_consensus.cc:513] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.570022 981 ts_tablet_manager.cc:1939] T 9a0bbafe1ea34ab5ae1b3de25b50d4a3 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:42.570151 1789 leader_election.cc:290] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.555670 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5af1dea3c6ca4f498ce6c039d72a14b7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.571468 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet 1be8f8557b9a4ae2917c37b8d01cedf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:42 UTC) from {username='slave'} at 127.0.0.1:35542
I20250623 08:04:42.574882 1902 consensus_queue.cc:1035] T 45293db3c40b4e4194445f12e7475e66 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:42.583137 979 log.cc:1199] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/501b5233522f43c7aed3c9770adb5c03
I20250623 08:04:42.583595 979 ts_tablet_manager.cc:1939] T 501b5233522f43c7aed3c9770adb5c03 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:42.584542 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 385) took 1349 ms. Trace:
I20250623 08:04:42.584688 979 rpcz_store.cc:276] 0623 08:04:41.234980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.235029 (+ 49us) service_pool.cc:225] Handling call
0623 08:04:42.098003 (+862974us) ts_tablet_manager.cc:1145] Deleting tablet 501b5233522f43c7aed3c9770adb5c03
0623 08:04:42.098011 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.329796 (+231785us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.511422 (+181626us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.584517 (+ 73095us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":862809}
I20250623 08:04:42.585166 979 tablet_replica.cc:331] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:42.585546 979 raft_consensus.cc:2241] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:42.586241 979 raft_consensus.cc:2270] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.587792 979 ts_tablet_manager.cc:1905] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:42.592562 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 44986.9us
I20250623 08:04:42.593817 32453 leader_election.cc:304] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.594134 1953 raft_consensus.cc:2802] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.594345 1953 raft_consensus.cc:695] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:42.594604 1953 consensus_queue.cc:237] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.597007 32314 leader_election.cc:304] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:42.597441 1843 raft_consensus.cc:2747] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:42.598331 1843 consensus_queue.cc:1035] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:42.599288 32314 consensus_peers.cc:487] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4a96ae9190640dea11bd11a9f0e4372. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:42.599932 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 271) took 5712 ms. Trace:
I20250623 08:04:42.600075 981 rpcz_store.cc:276] 0623 08:04:36.887175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.888029 (+ 854us) service_pool.cc:225] Handling call
0623 08:04:42.131907 (+5243878us) ts_tablet_manager.cc:1145] Deleting tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3
0623 08:04:42.131917 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.449862 (+317945us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.569307 (+119445us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.599913 (+ 30606us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5243679}
I20250623 08:04:42.600625 981 tablet_replica.cc:331] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:42.600854 981 raft_consensus.cc:2241] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:42.601063 981 raft_consensus.cc:2270] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.601611 981 ts_tablet_manager.cc:1905] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.556846 1823 raft_consensus.cc:513] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.611603 342 tablet_bootstrap.cc:492] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.611807 342 ts_tablet_manager.cc:1397] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.008s
I20250623 08:04:42.612500 342 raft_consensus.cc:357] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.612722 342 raft_consensus.cc:383] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.612823 342 raft_consensus.cc:738] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.613044 342 consensus_queue.cc:260] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.613483 342 ts_tablet_manager.cc:1428] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:42.613803 342 tablet_bootstrap.cc:492] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.615525 342 tablet_bootstrap.cc:654] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.618654 1359 raft_consensus.cc:491] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.618851 1359 raft_consensus.cc:513] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.619385 1359 leader_election.cc:290] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.626914 1627 raft_consensus.cc:491] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.627063 1627 raft_consensus.cc:513] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.627622 1627 leader_election.cc:290] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.633926 1843 raft_consensus.cc:491] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.634090 1843 raft_consensus.cc:513] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.634600 1843 leader_election.cc:290] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.640365 339 tablet_bootstrap.cc:492] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:42.640630 339 ts_tablet_manager.cc:1397] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.212s user 0.010s sys 0.000s
I20250623 08:04:42.641641 339 raft_consensus.cc:357] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.641945 339 raft_consensus.cc:383] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.642059 339 raft_consensus.cc:738] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.642424 339 consensus_queue.cc:260] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.644352 1823 leader_election.cc:290] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:42.647147 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 53351.5us
I20250623 08:04:42.647369 32453 leader_election.cc:304] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.647616 32453 leader_election.cc:304] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.647850 32453 leader_election.cc:304] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.648082 1359 raft_consensus.cc:2802] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.648233 1359 raft_consensus.cc:695] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:42.648468 1359 consensus_queue.cc:237] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.649387 1627 raft_consensus.cc:2802] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.649484 1627 raft_consensus.cc:491] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.649565 1627 raft_consensus.cc:3058] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.652238 1627 raft_consensus.cc:513] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.652726 1627 leader_election.cc:290] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.652873 1627 raft_consensus.cc:2802] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.652935 339 ts_tablet_manager.cc:1428] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250623 08:04:42.652954 1627 raft_consensus.cc:491] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.653040 1627 raft_consensus.cc:3058] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.653244 339 tablet_bootstrap.cc:492] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:42.654590 339 tablet_bootstrap.cc:654] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.655614 1627 raft_consensus.cc:513] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.656170 1627 leader_election.cc:290] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:42.662055 32314 leader_election.cc:343] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 3249605158634f4eab5b9d61357c383f
W20250623 08:04:42.662317 32314 leader_election.cc:343] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 3249605158634f4eab5b9d61357c383f
I20250623 08:04:42.662497 32314 leader_election.cc:304] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:42.662818 1843 raft_consensus.cc:2747] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:42.596186 1902 consensus_queue.cc:1035] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:42.596392 32589 leader_election.cc:304] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:42.642576 343 tablet_bootstrap.cc:492] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.666354 343 ts_tablet_manager.cc:1397] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.232s user 0.016s sys 0.028s
W20250623 08:04:42.666440 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 70065.2us
I20250623 08:04:42.666967 1902 raft_consensus.cc:2802] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.667074 343 raft_consensus.cc:357] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.667177 1902 raft_consensus.cc:695] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:42.667217 343 raft_consensus.cc:383] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.667263 343 raft_consensus.cc:738] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.667421 343 consensus_queue.cc:260] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.667572 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6fdea59f71e4a9d9fca8ede4b9439d2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.667886 343 ts_tablet_manager.cc:1428] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250623 08:04:42.668146 343 tablet_bootstrap.cc:492] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.669516 343 tablet_bootstrap.cc:654] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:42.669723 32589 consensus_peers.cc:487] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:42.670324 32589 leader_election.cc:343] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.667421 1902 consensus_queue.cc:237] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.684674 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16bc969d09f74b0986070a504f9f670e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:42.684865 32651 raft_consensus.cc:3058] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.685041 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c5ecd4fa5574a1b90b6e82f89ee88f7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:42.685233 32649 raft_consensus.cc:3058] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.687659 32651 raft_consensus.cc:2466] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.687978 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d1ec9d56ba40128325a50f204fe920" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:42.688043 32649 raft_consensus.cc:2466] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.688105 32656 raft_consensus.cc:3058] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.690886 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c14e325fa5f4f848748c346bc25dde0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.691056 32646 raft_consensus.cc:2466] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.692337 32656 raft_consensus.cc:2466] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.694190 343 tablet_bootstrap.cc:492] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.694317 343 ts_tablet_manager.cc:1397] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.005s
I20250623 08:04:42.695012 343 raft_consensus.cc:357] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.695163 343 raft_consensus.cc:383] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.695214 343 raft_consensus.cc:738] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.697145 343 consensus_queue.cc:260] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.697702 343 ts_tablet_manager.cc:1428] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:42.697938 343 tablet_bootstrap.cc:492] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.699327 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c5ecd4fa5574a1b90b6e82f89ee88f7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.700322 343 tablet_bootstrap.cc:654] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.711733 343 tablet_bootstrap.cc:492] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.711886 343 ts_tablet_manager.cc:1397] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20250623 08:04:42.712628 343 raft_consensus.cc:357] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.712783 343 raft_consensus.cc:383] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.712839 343 raft_consensus.cc:738] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.712801 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bfa787f828b411ab68d81bb896f7fb6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.713035 343 consensus_queue.cc:260] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.713598 343 ts_tablet_manager.cc:1428] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:42.713825 343 tablet_bootstrap.cc:492] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.715317 343 tablet_bootstrap.cc:654] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.712077 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16bc969d09f74b0986070a504f9f670e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.717315 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0d1ec9d56ba40128325a50f204fe920" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.717552 32373 raft_consensus.cc:3058] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.720840 32373 raft_consensus.cc:2466] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.721649 32373 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c14e325fa5f4f848748c346bc25dde0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.725612 343 tablet_bootstrap.cc:492] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.725754 343 ts_tablet_manager.cc:1397] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20250623 08:04:42.726438 343 raft_consensus.cc:357] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.726662 343 raft_consensus.cc:383] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.726727 343 raft_consensus.cc:738] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.726954 343 consensus_queue.cc:260] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.727315 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9f4829e103b460eb582beff3b966bb4" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:42.728111 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6fdea59f71e4a9d9fca8ede4b9439d2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:42.728317 32514 raft_consensus.cc:3058] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.731627 32514 raft_consensus.cc:2466] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:42.732220 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bfa787f828b411ab68d81bb896f7fb6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.733870 339 tablet_bootstrap.cc:492] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:42.734082 339 ts_tablet_manager.cc:1397] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.000s
I20250623 08:04:42.735118 343 ts_tablet_manager.cc:1428] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250623 08:04:42.735064 339 raft_consensus.cc:357] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.735322 339 raft_consensus.cc:383] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.735406 343 tablet_bootstrap.cc:492] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.735420 339 raft_consensus.cc:738] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.736173 339 consensus_queue.cc:260] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.737030 339 ts_tablet_manager.cc:1428] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:42.737427 339 tablet_bootstrap.cc:492] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:42.739334 339 tablet_bootstrap.cc:654] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.740239 343 tablet_bootstrap.cc:654] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.740489 342 tablet_bootstrap.cc:492] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.740635 342 ts_tablet_manager.cc:1397] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.127s user 0.024s sys 0.028s
I20250623 08:04:42.741334 342 raft_consensus.cc:357] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.741503 342 raft_consensus.cc:383] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.741569 342 raft_consensus.cc:738] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.741782 342 consensus_queue.cc:260] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.742249 342 ts_tablet_manager.cc:1428] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:42.742530 342 tablet_bootstrap.cc:492] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.744084 342 tablet_bootstrap.cc:654] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.758354 342 tablet_bootstrap.cc:492] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.758517 342 ts_tablet_manager.cc:1397] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20250623 08:04:42.759295 342 raft_consensus.cc:357] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.759459 342 raft_consensus.cc:383] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.759537 342 raft_consensus.cc:738] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.759768 342 consensus_queue.cc:260] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.760258 342 ts_tablet_manager.cc:1428] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:42.760550 342 tablet_bootstrap.cc:492] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.762423 342 tablet_bootstrap.cc:654] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:42.763020 32453 leader_election.cc:343] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:42.763715 32453 leader_election.cc:343] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.767647 342 tablet_bootstrap.cc:492] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.767787 342 ts_tablet_manager.cc:1397] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250623 08:04:42.768671 952 ts_tablet_manager.cc:1918] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:42.768800 952 log.cc:1199] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b81263a9ee9d4da1ac721c3b2bd21248
I20250623 08:04:42.769188 952 ts_tablet_manager.cc:1939] T b81263a9ee9d4da1ac721c3b2bd21248 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:42.790297 342 raft_consensus.cc:357] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.790514 342 raft_consensus.cc:383] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.790588 342 raft_consensus.cc:738] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.790822 342 consensus_queue.cc:260] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.791388 342 ts_tablet_manager.cc:1428] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20250623 08:04:42.791702 342 tablet_bootstrap.cc:492] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:42.792052 32453 leader_election.cc:343] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.792755 32453 leader_election.cc:304] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:42.792987 1627 raft_consensus.cc:2802] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.793097 1627 raft_consensus.cc:695] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:42.793310 1627 consensus_queue.cc:237] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.793313 342 tablet_bootstrap.cc:654] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.801208 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 243) took 6059 ms. Trace:
I20250623 08:04:42.801913 952 rpcz_store.cc:276] 0623 08:04:36.741886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.741930 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:42.411682 (+5669752us) ts_tablet_manager.cc:1145] Deleting tablet b81263a9ee9d4da1ac721c3b2bd21248
0623 08:04:42.411692 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.590175 (+178483us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.768537 (+178362us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.801153 (+ 32616us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32648,"tablet-delete.queue_time_us":5656798}
I20250623 08:04:42.802632 952 tablet_replica.cc:331] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:42.745117 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f9f4829e103b460eb582beff3b966bb4" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.803581 32378 raft_consensus.cc:3058] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.804208 979 ts_tablet_manager.cc:1918] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:42.804307 979 log.cc:1199] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/f9f4829e103b460eb582beff3b966bb4
I20250623 08:04:42.804797 952 raft_consensus.cc:2241] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:42.805124 952 raft_consensus.cc:2270] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:42.805574 32453 leader_election.cc:343] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.805665 952 ts_tablet_manager.cc:1905] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.806180 32378 raft_consensus.cc:2466] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.746001 1682 raft_consensus.cc:491] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.806700 1682 raft_consensus.cc:513] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.807236 1682 leader_election.cc:290] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.749290 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
W20250623 08:04:42.749593 32314 consensus_peers.cc:487] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6926c49b787941d98c647543bd3b9ed5. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:42.813189 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 63617.8us
I20250623 08:04:42.814888 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37b38cb54d6c45308358f556a692eabe" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.815888 342 tablet_bootstrap.cc:492] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.816046 342 ts_tablet_manager.cc:1397] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.006s
I20250623 08:04:42.818809 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37b38cb54d6c45308358f556a692eabe" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.819181 2012 raft_consensus.cc:491] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.819175 342 raft_consensus.cc:357] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.819365 342 raft_consensus.cc:383] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.819446 342 raft_consensus.cc:738] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.819470 2012 raft_consensus.cc:513] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.819720 1627 raft_consensus.cc:491] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.819911 342 consensus_queue.cc:260] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.819911 1627 raft_consensus.cc:513] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.820468 1627 leader_election.cc:290] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.820557 342 ts_tablet_manager.cc:1428] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:42.820602 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5443f12974bf41929d9cd937f146cd56" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:42.820763 32453 leader_election.cc:343] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.820891 342 tablet_bootstrap.cc:492] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:42.821054 32453 leader_election.cc:343] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:42.821192 32589 leader_election.cc:343] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.821511 2012 leader_election.cc:290] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:42.821971 32453 leader_election.cc:343] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:42.822623 32453 leader_election.cc:343] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.822865 342 tablet_bootstrap.cc:654] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.823133 1627 consensus_queue.cc:1035] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:42.823523 32025 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 49782.4us
I20250623 08:04:42.823904 32314 leader_election.cc:304] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:42.824153 1682 raft_consensus.cc:2747] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:42.791354 981 ts_tablet_manager.cc:1918] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:42.827256 981 log.cc:1199] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/205da880685b49b3b8669f73dc2c563b
I20250623 08:04:42.827289 952 ts_tablet_manager.cc:1918] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:42.827476 952 log.cc:1199] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/20e7a78cfc384033810603316b9e66bf
I20250623 08:04:42.827697 981 ts_tablet_manager.cc:1939] T 205da880685b49b3b8669f73dc2c563b P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:42.827865 952 ts_tablet_manager.cc:1939] T 20e7a78cfc384033810603316b9e66bf P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:42.828004 1902 raft_consensus.cc:491] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.828114 342 tablet_bootstrap.cc:492] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.828138 1902 raft_consensus.cc:513] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.828256 342 ts_tablet_manager.cc:1397] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20250623 08:04:42.828631 1902 leader_election.cc:290] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.828945 342 raft_consensus.cc:357] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.829118 342 raft_consensus.cc:383] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.829196 342 raft_consensus.cc:738] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.829399 342 consensus_queue.cc:260] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.829792 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90037c523a1f4118af268e088bb97062" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.829890 342 ts_tablet_manager.cc:1428] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:42.830183 342 tablet_bootstrap.cc:492] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.830288 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 259) took 5370 ms. Trace:
W20250623 08:04:42.830358 32589 leader_election.cc:343] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.791728 32380 raft_consensus.cc:3058] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.830437 952 rpcz_store.cc:276] 0623 08:04:37.459328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.460002 (+ 674us) service_pool.cc:225] Handling call
0623 08:04:42.802560 (+5342558us) ts_tablet_manager.cc:1145] Deleting tablet 20e7a78cfc384033810603316b9e66bf
0623 08:04:42.802569 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.808633 (+ 6064us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.827156 (+ 18523us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.830242 (+ 3086us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5342390}
I20250623 08:04:42.830893 952 tablet_replica.cc:331] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:42.831151 952 raft_consensus.cc:2241] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:42.832726 952 raft_consensus.cc:2270] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.833305 952 ts_tablet_manager.cc:1905] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.835659 343 tablet_bootstrap.cc:492] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.831796 342 tablet_bootstrap.cc:654] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.835793 343 ts_tablet_manager.cc:1397] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.034s
I20250623 08:04:42.836550 343 raft_consensus.cc:357] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.836728 343 raft_consensus.cc:383] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.836787 343 raft_consensus.cc:738] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.836974 343 consensus_queue.cc:260] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.837400 343 ts_tablet_manager.cc:1428] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:42.837661 343 tablet_bootstrap.cc:492] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.839335 343 tablet_bootstrap.cc:654] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.839830 979 ts_tablet_manager.cc:1939] T f9f4829e103b460eb582beff3b966bb4 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:42.843250 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 389) took 1607 ms. Trace:
I20250623 08:04:42.843387 979 rpcz_store.cc:276] 0623 08:04:41.236091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.236134 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:42.585119 (+1348985us) ts_tablet_manager.cc:1145] Deleting tablet f9f4829e103b460eb582beff3b966bb4
0623 08:04:42.585128 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.706240 (+121112us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.804100 (+ 97860us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.843231 (+ 39131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348839}
I20250623 08:04:42.843887 979 tablet_replica.cc:331] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:42.844105 979 raft_consensus.cc:2241] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:42.844290 979 raft_consensus.cc:2270] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.844828 979 ts_tablet_manager.cc:1905] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.845009 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 267) took 6033 ms. Trace:
I20250623 08:04:42.845098 981 rpcz_store.cc:276] 0623 08:04:36.811469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.877861 (+ 66392us) service_pool.cc:225] Handling call
0623 08:04:42.600576 (+5722715us) ts_tablet_manager.cc:1145] Deleting tablet 205da880685b49b3b8669f73dc2c563b
0623 08:04:42.600587 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.723533 (+122946us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.791236 (+ 67703us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.844991 (+ 53755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5711604}
I20250623 08:04:42.845388 981 tablet_replica.cc:331] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:42.845659 981 raft_consensus.cc:2241] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:42.846067 981 raft_consensus.cc:2270] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.846664 981 ts_tablet_manager.cc:1905] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:42.847486 32453 leader_election.cc:343] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:42.847755 32453 leader_election.cc:343] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.847990 32453 leader_election.cc:304] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:42.848474 32453 leader_election.cc:343] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.848594 1627 raft_consensus.cc:2802] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.848840 1627 raft_consensus.cc:695] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
W20250623 08:04:42.848784 32453 leader_election.cc:343] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:42.849162 32453 consensus_peers.cc:487] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:42.849116 1627 consensus_queue.cc:237] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.849504 32453 leader_election.cc:304] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:42.849799 1359 raft_consensus.cc:2802] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.849947 1359 raft_consensus.cc:695] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:42.850183 1359 consensus_queue.cc:237] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.850244 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1be8f8557b9a4ae2917c37b8d01cedf6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.850566 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1be8f8557b9a4ae2917c37b8d01cedf6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:42.850728 32649 raft_consensus.cc:3058] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.854178 32649 raft_consensus.cc:2466] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.856671 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e17e1ca404b462da43aa1db7dc3b556" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:42.856938 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e17e1ca404b462da43aa1db7dc3b556" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:42.857098 32649 raft_consensus.cc:3058] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.859786 32649 raft_consensus.cc:2466] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:42.848804 952 ts_tablet_manager.cc:1918] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:42.862102 952 log.cc:1199] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/136e482995a24c5daec55a701f3091fa
I20250623 08:04:42.862689 1902 consensus_queue.cc:1035] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:42.862808 952 ts_tablet_manager.cc:1939] T 136e482995a24c5daec55a701f3091fa P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:42.863571 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 260) took 5404 ms. Trace:
I20250623 08:04:42.863534 32589 leader_election.cc:304] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:42.863780 1902 raft_consensus.cc:2802] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.863682 952 rpcz_store.cc:276] 0623 08:04:37.459432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.460195 (+ 763us) service_pool.cc:225] Handling call
0623 08:04:42.830818 (+5370623us) ts_tablet_manager.cc:1145] Deleting tablet 136e482995a24c5daec55a701f3091fa
0623 08:04:42.830828 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.840905 (+ 10077us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.848674 (+ 7769us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.863534 (+ 14860us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5370450}
I20250623 08:04:42.863895 1902 raft_consensus.cc:695] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:42.863999 952 tablet_replica.cc:331] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:42.864198 952 raft_consensus.cc:2241] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:42.864128 1902 consensus_queue.cc:237] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.864446 952 raft_consensus.cc:2270] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.865007 952 ts_tablet_manager.cc:1905] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.863525 343 tablet_bootstrap.cc:492] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.870476 343 ts_tablet_manager.cc:1397] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250623 08:04:42.875389 979 ts_tablet_manager.cc:1918] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:42.878520 342 tablet_bootstrap.cc:492] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.878701 342 ts_tablet_manager.cc:1397] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.002s
I20250623 08:04:42.884100 342 raft_consensus.cc:357] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.884306 342 raft_consensus.cc:383] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.884384 342 raft_consensus.cc:738] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.890141 342 consensus_queue.cc:260] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.891305 342 ts_tablet_manager.cc:1428] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250623 08:04:42.891602 342 tablet_bootstrap.cc:492] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.898229 342 tablet_bootstrap.cc:654] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.900543 952 ts_tablet_manager.cc:1918] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250623 08:04:42.904273 339 tablet_bootstrap.cc:492] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:42.904456 339 ts_tablet_manager.cc:1397] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.167s user 0.005s sys 0.000s
I20250623 08:04:42.905114 339 raft_consensus.cc:357] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.905273 339 raft_consensus.cc:383] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.905321 339 raft_consensus.cc:738] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.902541 342 tablet_bootstrap.cc:492] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.905546 342 ts_tablet_manager.cc:1397] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20250623 08:04:42.906374 342 raft_consensus.cc:357] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.908259 342 raft_consensus.cc:383] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.908458 342 raft_consensus.cc:738] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.910732 342 consensus_queue.cc:260] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.911775 342 ts_tablet_manager.cc:1428] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250623 08:04:42.915257 342 tablet_bootstrap.cc:492] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.915927 2046 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d31835ec4ad74332a00998d431492466)
I20250623 08:04:42.917100 2046 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:42.918164 2046 leader_election.cc:290] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313)
I20250623 08:04:42.929852 2055 tablet.cc:1722] T f8a5e35c1974456b8079ab8907c2652e P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:42.930807 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8945b6f49474648ac73bcb85285680a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:42.931447 339 consensus_queue.cc:260] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.931784 342 tablet_bootstrap.cc:654] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.932081 339 ts_tablet_manager.cc:1428] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250623 08:04:42.932375 339 tablet_bootstrap.cc:492] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:42.932670 32453 leader_election.cc:304] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:42.932974 1823 raft_consensus.cc:2802] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.933082 1823 raft_consensus.cc:695] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:42.933300 1823 consensus_queue.cc:237] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.933637 339 tablet_bootstrap.cc:654] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.935330 1627 raft_consensus.cc:491] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.935495 1627 raft_consensus.cc:513] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.936056 1627 leader_election.cc:290] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.940201 32453 leader_election.cc:304] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:42.940439 1823 raft_consensus.cc:2802] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.940585 1823 raft_consensus.cc:695] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:42.940814 1823 consensus_queue.cc:237] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.945569 2047 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d31835ec4ad74332a00998d431492466)
I20250623 08:04:42.945658 32453 leader_election.cc:304] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:42.945901 1627 raft_consensus.cc:2802] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.945894 2047 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:42.946009 1627 raft_consensus.cc:491] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.946101 1627 raft_consensus.cc:3058] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:42.946143 32453 leader_election.cc:343] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:42.946245 32453 leader_election.cc:304] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:42.946604 1957 raft_consensus.cc:2747] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:42.947299 457 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9752155c20ee42fbb7ff7b7d5b682444" candidate_term: 4 candidate_status { last_received { term: 3 index: 760 } } ignore_live_leader: false dest_uuid: "6644af0f4b2a4b998282b021ec1dc678" is_pre_election: true
I20250623 08:04:42.947520 457 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9752155c20ee42fbb7ff7b7d5b682444 in term 3.
I20250623 08:04:42.948030 1213 leader_election.cc:304] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678, 9752155c20ee42fbb7ff7b7d5b682444; no voters:
I20250623 08:04:42.949229 1627 raft_consensus.cc:513] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.950004 1627 leader_election.cc:290] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:42.950826 342 tablet_bootstrap.cc:492] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.950992 342 ts_tablet_manager.cc:1397] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20250623 08:04:42.907897 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8945b6f49474648ac73bcb85285680a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:42.951265 32649 raft_consensus.cc:2466] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:42.914149 343 raft_consensus.cc:357] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.951553 343 raft_consensus.cc:383] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.951628 343 raft_consensus.cc:738] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.951853 343 consensus_queue.cc:260] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.952386 343 ts_tablet_manager.cc:1428] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.082s user 0.002s sys 0.000s
I20250623 08:04:42.952688 343 tablet_bootstrap.cc:492] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.953176 342 raft_consensus.cc:357] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.953441 342 raft_consensus.cc:383] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.953553 342 raft_consensus.cc:738] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.953869 342 consensus_queue.cc:260] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.954496 342 ts_tablet_manager.cc:1428] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:42.954813 342 tablet_bootstrap.cc:492] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:42.956629 342 tablet_bootstrap.cc:654] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.957125 343 tablet_bootstrap.cc:654] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:42.958020 979 log.cc:1199] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b84b8b8713774ab4896842feb9fb075e
I20250623 08:04:42.958385 979 ts_tablet_manager.cc:1939] T b84b8b8713774ab4896842feb9fb075e P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:42.951064 32453 leader_election.cc:343] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: Tablet not RUNNING: STOPPED
I20250623 08:04:42.960338 32453 leader_election.cc:304] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.960616 32453 leader_election.cc:304] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.961014 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 388) took 1725 ms. Trace:
I20250623 08:04:42.961149 979 rpcz_store.cc:276] 0623 08:04:41.235996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.236302 (+ 306us) service_pool.cc:225] Handling call
0623 08:04:42.843845 (+1607543us) ts_tablet_manager.cc:1145] Deleting tablet b84b8b8713774ab4896842feb9fb075e
0623 08:04:42.843854 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.872067 (+ 28213us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.875297 (+ 3230us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.960991 (+ 85694us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1607431}
I20250623 08:04:42.963061 979 tablet_replica.cc:331] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:42.963299 979 raft_consensus.cc:2241] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:42.963618 979 raft_consensus.cc:2270] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:42.964514 2047 leader_election.cc:290] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:42.966394 1957 raft_consensus.cc:2802] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.966521 1957 raft_consensus.cc:695] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:42.966604 1823 raft_consensus.cc:2802] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:42.966740 2047 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250623 08:04:42.966797 1957 consensus_queue.cc:237] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.966945 1823 raft_consensus.cc:695] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:42.967195 2047 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d31835ec4ad74332a00998d431492466)
I20250623 08:04:42.967211 1823 consensus_queue.cc:237] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.967537 2047 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 3 FOLLOWER]: Advancing to term 4
I20250623 08:04:42.973474 1359 consensus_queue.cc:1035] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:42.978928 979 ts_tablet_manager.cc:1905] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:42.981459 339 tablet_bootstrap.cc:492] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:42.981613 339 ts_tablet_manager.cc:1397] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.000s
I20250623 08:04:42.982111 952 log.cc:1199] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/283bebc7063d437ba3e541d7dee3c0b5
I20250623 08:04:42.983608 339 raft_consensus.cc:357] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.983786 339 raft_consensus.cc:383] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.983837 339 raft_consensus.cc:738] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.984014 339 consensus_queue.cc:260] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:42.984827 343 tablet_bootstrap.cc:492] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:42.984969 343 ts_tablet_manager.cc:1397] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.001s
I20250623 08:04:42.985626 343 raft_consensus.cc:357] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.985782 343 raft_consensus.cc:383] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.985903 343 raft_consensus.cc:738] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.986137 343 consensus_queue.cc:260] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.986752 343 ts_tablet_manager.cc:1428] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:42.987319 339 ts_tablet_manager.cc:1428] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:42.987633 343 tablet_bootstrap.cc:492] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:42.988619 2014 raft_consensus.cc:491] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:42.988727 2014 raft_consensus.cc:513] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.989252 2014 leader_election.cc:290] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:42.989506 981 ts_tablet_manager.cc:1918] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:42.989610 981 log.cc:1199] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/e9f8aa424bbb4aa8a907259270d46656
I20250623 08:04:42.990006 981 ts_tablet_manager.cc:1939] T e9f8aa424bbb4aa8a907259270d46656 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:42.990149 952 ts_tablet_manager.cc:1939] T 283bebc7063d437ba3e541d7dee3c0b5 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:42.984982 32453 leader_election.cc:304] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:42.990352 1267 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 4 candidate_status { last_received { term: 3 index: 760 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
I20250623 08:04:42.990826 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5443f12974bf41929d9cd937f146cd56" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.990983 32516 raft_consensus.cc:2466] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:42.991142 1957 raft_consensus.cc:2802] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:42.991240 1957 raft_consensus.cc:491] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:42.991317 1957 raft_consensus.cc:3058] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:42.983671 342 tablet_bootstrap.cc:492] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:42.994642 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90037c523a1f4118af268e088bb97062" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.994872 32518 raft_consensus.cc:2466] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:42.994910 342 ts_tablet_manager.cc:1397] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20250623 08:04:42.995790 342 raft_consensus.cc:357] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:42.995999 342 raft_consensus.cc:383] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:42.996073 342 raft_consensus.cc:738] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:42.998138 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0e9b2f810014e81a29ca9b41cd3f8ba" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:42.998296 32519 raft_consensus.cc:2466] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:42.998692 2047 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:42.999827 1957 raft_consensus.cc:513] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.000373 1957 leader_election.cc:290] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.000626 342 consensus_queue.cc:260] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.001140 342 ts_tablet_manager.cc:1428] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:43.001431 342 tablet_bootstrap.cc:492] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.005394 32589 leader_election.cc:304] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.006132 32589 leader_election.cc:304] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.006567 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c14e325fa5f4f848748c346bc25dde0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.006778 32655 raft_consensus.cc:3058] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.006975 1902 raft_consensus.cc:2802] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.007021 1957 raft_consensus.cc:491] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.007063 1902 raft_consensus.cc:491] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.007133 1902 raft_consensus.cc:3058] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.007128 1957 raft_consensus.cc:513] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.007630 1957 leader_election.cc:290] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.008419 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 261) took 5548 ms. Trace:
I20250623 08:04:43.008496 952 rpcz_store.cc:276] 0623 08:04:37.459520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.460899 (+ 1379us) service_pool.cc:225] Handling call
0623 08:04:42.863935 (+5403036us) ts_tablet_manager.cc:1145] Deleting tablet 283bebc7063d437ba3e541d7dee3c0b5
0623 08:04:42.863946 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.881781 (+ 17835us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.900423 (+ 18642us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.008385 (+107962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5402897}
I20250623 08:04:43.008880 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 263) took 5549 ms. Trace:
I20250623 08:04:43.008945 952 rpcz_store.cc:276] 0623 08:04:37.459672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.461076 (+ 1404us) service_pool.cc:225] Handling call
0623 08:04:43.008770 (+5547694us) ts_tablet_manager.cc:1145] Deleting tablet a34da529ffde4595a23fc6596b30e3e9
0623 08:04:43.008779 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.008866 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5547518}
I20250623 08:04:43.009168 952 tablet_replica.cc:331] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.009320 952 raft_consensus.cc:2241] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.009477 952 raft_consensus.cc:2270] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.009657 1902 raft_consensus.cc:513] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.010003 952 ts_tablet_manager.cc:1905] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.010165 1902 leader_election.cc:290] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.010196 32655 raft_consensus.cc:2466] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.010453 1902 raft_consensus.cc:2802] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.010514 1902 raft_consensus.cc:491] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.010568 1902 raft_consensus.cc:3058] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.011101 32589 leader_election.cc:304] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:43.011741 32453 leader_election.cc:343] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.011898 2012 raft_consensus.cc:2802] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.011993 2012 raft_consensus.cc:491] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.012060 2012 raft_consensus.cc:3058] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:43.012058 32453 leader_election.cc:343] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.013810 1902 raft_consensus.cc:513] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.014571 2012 raft_consensus.cc:513] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.014840 1902 leader_election.cc:290] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.006160 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "affa3da7540648d28c5f00a3cf348eb1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.015460 32647 raft_consensus.cc:2466] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.017345 2012 leader_election.cc:290] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.018947 32370 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c14e325fa5f4f848748c346bc25dde0" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.019218 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5443f12974bf41929d9cd937f146cd56" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.019220 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90037c523a1f4118af268e088bb97062" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.019536 339 tablet_bootstrap.cc:492] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.019716 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "affa3da7540648d28c5f00a3cf348eb1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.018900 342 tablet_bootstrap.cc:654] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.021134 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0e9b2f810014e81a29ca9b41cd3f8ba" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:43.021798 32453 leader_election.cc:343] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:43.022534 32589 leader_election.cc:343] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:43.022681 32589 leader_election.cc:343] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:43.022907 32589 leader_election.cc:343] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:43.023109 32453 leader_election.cc:343] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:43.024180 32453 leader_election.cc:343] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.024361 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0e9b2f810014e81a29ca9b41cd3f8ba" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.025069 32589 leader_election.cc:343] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.024911 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 272) took 6136 ms. Trace:
I20250623 08:04:43.025328 981 rpcz_store.cc:276] 0623 08:04:36.888396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.889631 (+ 1235us) service_pool.cc:225] Handling call
0623 08:04:42.845347 (+5955716us) ts_tablet_manager.cc:1145] Deleting tablet e9f8aa424bbb4aa8a907259270d46656
0623 08:04:42.845357 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:42.849868 (+ 4511us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:42.989392 (+139524us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.024894 (+ 35502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5955564}
I20250623 08:04:43.025715 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8945b6f49474648ac73bcb85285680a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.025848 32646 raft_consensus.cc:3058] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.028609 32646 raft_consensus.cc:2466] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.025480 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8945b6f49474648ac73bcb85285680a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.029083 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5443f12974bf41929d9cd937f146cd56" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:43.029230 32519 raft_consensus.cc:3058] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.029263 981 tablet_replica.cc:331] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.029453 981 raft_consensus.cc:2241] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.029584 981 raft_consensus.cc:2270] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.029719 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "803d3607d75443568b7ddc9f0c0bdc90" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.029860 32647 raft_consensus.cc:2466] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.030038 981 ts_tablet_manager.cc:1905] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.031280 457 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9752155c20ee42fbb7ff7b7d5b682444" candidate_term: 4 candidate_status { last_received { term: 3 index: 760 } } ignore_live_leader: false dest_uuid: "6644af0f4b2a4b998282b021ec1dc678"
I20250623 08:04:43.031472 457 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 3 FOLLOWER]: Advancing to term 4
I20250623 08:04:43.031795 32519 raft_consensus.cc:2466] T 5443f12974bf41929d9cd937f146cd56 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:43.032454 1267 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6644af0f4b2a4b998282b021ec1dc678 in current term 4: Already voted for candidate 9752155c20ee42fbb7ff7b7d5b682444 in this term.
I20250623 08:04:43.034811 2047 leader_election.cc:290] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 4 election: Requested vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 6644af0f4b2a4b998282b021ec1dc678 (127.31.69.60:39691)
I20250623 08:04:43.034889 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90037c523a1f4118af268e088bb97062" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:43.035045 32519 raft_consensus.cc:3058] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.030241 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "803d3607d75443568b7ddc9f0c0bdc90" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.037670 32519 raft_consensus.cc:2466] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:43.038061 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0e9b2f810014e81a29ca9b41cd3f8ba" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:43.038192 32519 raft_consensus.cc:3058] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.024374 339 tablet_bootstrap.cc:654] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.038777 457 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9752155c20ee42fbb7ff7b7d5b682444 in term 4.
I20250623 08:04:43.039414 1213 leader_election.cc:304] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678, 9752155c20ee42fbb7ff7b7d5b682444; no voters:
I20250623 08:04:43.039768 2047 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 4 FOLLOWER]: Leader election won for term 4
I20250623 08:04:43.040125 2047 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [term 4 LEADER]: Becoming Leader. State: Replica: 9752155c20ee42fbb7ff7b7d5b682444, State: Running, Role: LEADER
I20250623 08:04:43.040692 2047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 760, Committed index: 760, Last appended: 3.760, Last appended by leader: 760, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:43.040824 32519 raft_consensus.cc:2466] T a0e9b2f810014e81a29ca9b41cd3f8ba P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:43.043603 32453 leader_election.cc:304] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.044565 342 tablet_bootstrap.cc:492] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.044565 339 tablet_bootstrap.cc:492] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.044720 342 ts_tablet_manager.cc:1397] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.002s
I20250623 08:04:43.045357 342 raft_consensus.cc:357] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.045511 342 raft_consensus.cc:383] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.045576 342 raft_consensus.cc:738] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.045779 342 consensus_queue.cc:260] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.046216 339 ts_tablet_manager.cc:1397] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20250623 08:04:43.046370 342 ts_tablet_manager.cc:1428] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:43.046741 342 tablet_bootstrap.cc:492] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.047856 32380 raft_consensus.cc:3058] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.048120 342 tablet_bootstrap.cc:654] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.048799 1823 raft_consensus.cc:2802] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.048897 1823 raft_consensus.cc:695] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.049101 1823 consensus_queue.cc:237] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.051220 343 tablet_bootstrap.cc:654] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.054929 32453 leader_election.cc:304] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.055480 1957 raft_consensus.cc:2802] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.055562 1957 raft_consensus.cc:491] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.055637 1957 raft_consensus.cc:3058] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.058460 1957 raft_consensus.cc:513] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.059044 1957 leader_election.cc:290] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.059500 1359 consensus_queue.cc:1035] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.062544 952 ts_tablet_manager.cc:1918] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250623 08:04:43.064275 339 raft_consensus.cc:357] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.064443 339 raft_consensus.cc:383] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.064491 339 raft_consensus.cc:738] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.064700 339 consensus_queue.cc:260] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.066252 2060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9752155c20ee42fbb7ff7b7d5b682444. Latest consensus state: current_term: 4 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:43.066540 2060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: LEADER
I20250623 08:04:43.075038 952 log.cc:1199] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/8c01121f2e644d6aa6e8ec228bf75be9
I20250623 08:04:43.075479 952 ts_tablet_manager.cc:1939] T 8c01121f2e644d6aa6e8ec228bf75be9 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.076494 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 264) took 5616 ms. Trace:
I20250623 08:04:43.059520 32453 leader_election.cc:304] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.076905 32453 leader_election.cc:304] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.077088 1823 raft_consensus.cc:2802] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.078022 1627 raft_consensus.cc:2802] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.078094 1627 raft_consensus.cc:491] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.078152 1627 raft_consensus.cc:3058] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.080632 1627 raft_consensus.cc:513] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.081054 343 tablet_bootstrap.cc:492] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.081168 1627 leader_election.cc:290] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.081462 1823 raft_consensus.cc:695] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.081755 1823 consensus_queue.cc:237] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.082706 343 ts_tablet_manager.cc:1397] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.000s
I20250623 08:04:43.082856 32589 leader_election.cc:304] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.083292 32589 leader_election.cc:304] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:43.083484 32453 leader_election.cc:343] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.083513 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "affa3da7540648d28c5f00a3cf348eb1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.083912 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "affa3da7540648d28c5f00a3cf348eb1" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.084081 32647 raft_consensus.cc:3058] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:43.084332 32453 leader_election.cc:343] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.085274 342 tablet_bootstrap.cc:492] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.085377 342 ts_tablet_manager.cc:1397] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250623 08:04:43.086192 342 raft_consensus.cc:357] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.086324 342 raft_consensus.cc:383] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.086373 342 raft_consensus.cc:738] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.086529 342 consensus_queue.cc:260] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:43.086830 32453 leader_election.cc:343] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.087443 339 ts_tablet_manager.cc:1428] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20250623 08:04:43.087493 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "803d3607d75443568b7ddc9f0c0bdc90" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.087580 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "803d3607d75443568b7ddc9f0c0bdc90" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.087735 339 tablet_bootstrap.cc:492] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.087831 32654 raft_consensus.cc:3058] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.088588 32647 raft_consensus.cc:2466] T affa3da7540648d28c5f00a3cf348eb1 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.090615 32654 raft_consensus.cc:2466] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.091761 981 ts_tablet_manager.cc:1918] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:43.091789 342 ts_tablet_manager.cc:1428] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:43.091877 981 log.cc:1199] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/8dc696236a15420e9666cca708068fb2
I20250623 08:04:43.092068 342 tablet_bootstrap.cc:492] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.092199 981 ts_tablet_manager.cc:1939] T 8dc696236a15420e9666cca708068fb2 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.092954 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 274) took 6202 ms. Trace:
I20250623 08:04:43.093045 981 rpcz_store.cc:276] 0623 08:04:36.890422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.890488 (+ 66us) service_pool.cc:225] Handling call
0623 08:04:43.029190 (+6138702us) ts_tablet_manager.cc:1145] Deleting tablet 8dc696236a15420e9666cca708068fb2
0623 08:04:43.029223 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.071248 (+ 42025us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.091651 (+ 20403us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.092934 (+ 1283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35200,"tablet-delete.queue_time_us":6138002}
I20250623 08:04:43.097559 32589 leader_election.cc:304] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.098167 339 tablet_bootstrap.cc:654] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.098359 1902 raft_consensus.cc:2802] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.098502 1902 raft_consensus.cc:695] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:43.098783 1902 consensus_queue.cc:237] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.099797 2013 raft_consensus.cc:2802] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.100068 2014 raft_consensus.cc:2802] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.100852 2014 raft_consensus.cc:695] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:43.101109 2014 consensus_queue.cc:237] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.101406 32453 leader_election.cc:304] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.102146 1957 raft_consensus.cc:2802] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.102346 1957 raft_consensus.cc:695] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.102578 2076 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250623 08:04:43.102598 1957 consensus_queue.cc:237] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.107908 32453 leader_election.cc:304] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.108107 2013 raft_consensus.cc:695] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:43.108472 2013 consensus_queue.cc:237] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.108860 1359 raft_consensus.cc:2802] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.108983 1359 raft_consensus.cc:695] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.109220 1359 consensus_queue.cc:237] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.111624 952 rpcz_store.cc:276] 0623 08:04:37.459752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.461280 (+ 1528us) service_pool.cc:225] Handling call
0623 08:04:43.009135 (+5547855us) ts_tablet_manager.cc:1145] Deleting tablet 8c01121f2e644d6aa6e8ec228bf75be9
0623 08:04:43.009141 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.033965 (+ 24824us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.062425 (+ 28460us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.076475 (+ 14050us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5547683}
I20250623 08:04:43.112555 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 265) took 5652 ms. Trace:
I20250623 08:04:43.113493 952 rpcz_store.cc:276] 0623 08:04:37.459825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.461485 (+ 1660us) service_pool.cc:225] Handling call
0623 08:04:43.112430 (+5650945us) ts_tablet_manager.cc:1145] Deleting tablet 6b04e38d3da44301a450fe1fb5e2ada6
0623 08:04:43.112441 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.112540 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5650824}
I20250623 08:04:43.114131 952 tablet_replica.cc:331] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.115231 952 raft_consensus.cc:2241] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.115892 952 pending_rounds.cc:70] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:43.116400 952 pending_rounds.cc:77] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727455923347456 op_type: NO_OP noop_request { }
I20250623 08:04:43.116734 952 raft_consensus.cc:2887] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:43.117002 952 raft_consensus.cc:2270] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.115013 342 tablet_bootstrap.cc:654] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:43.113359 32453 leader_election.cc:343] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.120167 343 raft_consensus.cc:357] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.120368 343 raft_consensus.cc:383] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.120455 343 raft_consensus.cc:738] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.120867 2067 tablet.cc:1722] T f8a5e35c1974456b8079ab8907c2652e P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:43.121553 343 consensus_queue.cc:260] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.121837 979 ts_tablet_manager.cc:1918] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.123304 979 log.cc:1199] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/789d1c20aa72480ab71123c799bbde4c
I20250623 08:04:43.123812 979 ts_tablet_manager.cc:1939] T 789d1c20aa72480ab71123c799bbde4c P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:43.125586 343 ts_tablet_manager.cc:1428] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20250623 08:04:43.125972 343 tablet_bootstrap.cc:492] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.127339 342 tablet_bootstrap.cc:492] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.127482 342 ts_tablet_manager.cc:1397] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250623 08:04:43.128105 342 raft_consensus.cc:357] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.128273 342 raft_consensus.cc:383] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.128324 342 raft_consensus.cc:738] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.128507 342 consensus_queue.cc:260] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.128968 342 ts_tablet_manager.cc:1428] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250623 08:04:43.129215 342 tablet_bootstrap.cc:492] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.130579 342 tablet_bootstrap.cc:654] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.133946 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 387) took 1898 ms. Trace:
I20250623 08:04:43.135010 32380 raft_consensus.cc:3058] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.136361 1627 raft_consensus.cc:491] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.136484 1627 raft_consensus.cc:513] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.137094 1627 leader_election.cc:290] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.137421 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db81250d1e164a7d94affd764fc0ff0c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.138645 952 ts_tablet_manager.cc:1905] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:43.138525 32453 leader_election.cc:343] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.139058 1627 consensus_queue.cc:1035] T 306c4021b5d94a4996de07c222a8bfd8 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.139345 339 tablet_bootstrap.cc:492] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.139482 339 ts_tablet_manager.cc:1397] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.001s
I20250623 08:04:43.140471 339 raft_consensus.cc:357] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.140625 339 raft_consensus.cc:383] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.140682 339 raft_consensus.cc:738] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.140965 339 consensus_queue.cc:260] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.141736 339 ts_tablet_manager.cc:1428] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:43.147193 339 tablet_bootstrap.cc:492] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.148909 339 tablet_bootstrap.cc:654] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.149720 981 tablet_replica.cc:331] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.150066 981 raft_consensus.cc:2241] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.137763 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db81250d1e164a7d94affd764fc0ff0c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:43.150776 32453 leader_election.cc:343] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.150905 32453 leader_election.cc:304] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.151103 1957 raft_consensus.cc:2747] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.151384 979 rpcz_store.cc:276] 0623 08:04:41.235914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.236452 (+ 538us) service_pool.cc:225] Handling call
0623 08:04:42.963015 (+1726563us) ts_tablet_manager.cc:1145] Deleting tablet 789d1c20aa72480ab71123c799bbde4c
0623 08:04:42.963024 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.002250 (+ 39226us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.094669 (+ 92419us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.133894 (+ 39225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8370,"tablet-delete.queue_time_us":1726425}
I20250623 08:04:43.152735 342 tablet_bootstrap.cc:492] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.152849 342 ts_tablet_manager.cc:1397] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20250623 08:04:43.153101 981 raft_consensus.cc:2270] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.153426 342 raft_consensus.cc:357] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.153649 342 raft_consensus.cc:383] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.153719 342 raft_consensus.cc:738] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.153923 342 consensus_queue.cc:260] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.154470 342 ts_tablet_manager.cc:1428] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:43.154724 342 tablet_bootstrap.cc:492] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.155406 981 ts_tablet_manager.cc:1905] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.156561 342 tablet_bootstrap.cc:654] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.158828 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 386) took 1923 ms. Trace:
I20250623 08:04:43.159011 979 rpcz_store.cc:276] 0623 08:04:41.235828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.236606 (+ 778us) service_pool.cc:225] Handling call
0623 08:04:43.154906 (+1918300us) ts_tablet_manager.cc:1145] Deleting tablet ba3d2fa01d574dd992dc904ac024ba3a
0623 08:04:43.158673 (+ 3767us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.158781 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1917350}
I20250623 08:04:43.159832 979 tablet_replica.cc:331] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:43.159132 1682 raft_consensus.cc:491] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.160030 1682 raft_consensus.cc:513] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:43.160684 1682 leader_election.cc:290] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:43.161617 32314 leader_election.cc:343] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: dc0dad440b494018bc0d6cfd40940bc1
W20250623 08:04:43.163482 32314 leader_election.cc:343] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: dc0dad440b494018bc0d6cfd40940bc1
I20250623 08:04:43.163599 32314 leader_election.cc:304] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.163990 1682 raft_consensus.cc:2747] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.168251 979 raft_consensus.cc:2241] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.168792 979 pending_rounds.cc:70] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:43.169076 979 pending_rounds.cc:77] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727450629943296 op_type: NO_OP noop_request { }
I20250623 08:04:43.169368 979 raft_consensus.cc:2887] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:43.167024 343 tablet_bootstrap.cc:654] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.171167 979 raft_consensus.cc:2270] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:43.175885 2083 consensus_queue.cc:1162] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "125745849172449ba4be8244d61fdcb3" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250623 08:04:43.179562 979 ts_tablet_manager.cc:1905] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.179674 1682 raft_consensus.cc:491] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.179847 1682 raft_consensus.cc:513] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:43.180503 1682 leader_election.cc:290] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.180824 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9049e3d9b39f420bb51221808409fc54" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.180914 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9049e3d9b39f420bb51221808409fc54" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.181618 32314 leader_election.cc:304] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.182083 1910 raft_consensus.cc:2747] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:43.184506 32453 consensus_peers.cc:487] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:43.186674 343 tablet_bootstrap.cc:492] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.187640 343 ts_tablet_manager.cc:1397] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.001s
I20250623 08:04:43.188587 343 raft_consensus.cc:357] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.189572 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]
I20250623 08:04:43.190466 339 tablet_bootstrap.cc:492] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.190470 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]
I20250623 08:04:43.190691 339 ts_tablet_manager.cc:1397] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.004s
I20250623 08:04:43.191272 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5]
I20250623 08:04:43.191571 339 raft_consensus.cc:357] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.191735 339 raft_consensus.cc:383] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.191787 339 raft_consensus.cc:738] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.192001 339 consensus_queue.cc:260] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.192510 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_4533668b8daa4ff3ad00ccc95e97e032 [id=08d03b1b488c4b52abebcef8b3d0f853]
I20250623 08:04:43.192651 339 ts_tablet_manager.cc:1428] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:43.192927 339 tablet_bootstrap.cc:492] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.194736 339 tablet_bootstrap.cc:654] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.195789 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]
I20250623 08:04:43.198139 342 tablet_bootstrap.cc:492] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.198292 342 ts_tablet_manager.cc:1397] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.003s
I20250623 08:04:43.198478 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_4903e809cc2d447fb94224ccbb05078d [id=0b8ff648531f42dd94998cc01b79bf3e]
I20250623 08:04:43.199117 342 raft_consensus.cc:357] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.199313 342 raft_consensus.cc:383] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.199378 342 raft_consensus.cc:738] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.199635 342 consensus_queue.cc:260] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.200182 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]
I20250623 08:04:43.200325 342 ts_tablet_manager.cc:1428] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:43.200539 981 ts_tablet_manager.cc:1918] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:43.200613 342 tablet_bootstrap.cc:492] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.200789 343 raft_consensus.cc:383] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.202258 342 tablet_bootstrap.cc:654] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.207995 343 raft_consensus.cc:738] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.208413 343 consensus_queue.cc:260] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.211897 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_9968742b8af441aa8db7bd5343fb4059 [id=0e357a45fc4547d59f1f886531c3ac2c]
I20250623 08:04:43.212409 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_217651efa5dc4571b2a47cd3196c9e57 [id=0ea2aee143054b39ae96e30c64a42a50]
I20250623 08:04:43.212885 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_fb578541a1094c0f90ba5281797a457d [id=0ed6580f5bc04a48830d33c64804a1ee]
I20250623 08:04:43.213193 952 ts_tablet_manager.cc:1918] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.213315 952 log.cc:1199] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/771ecfc41e2b4007a4d213cec5b128c6
I20250623 08:04:43.213646 952 ts_tablet_manager.cc:1939] T 771ecfc41e2b4007a4d213cec5b128c6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.214296 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1631659a168049fba9411207edd8b56f [id=10d3dacba1db466893ef47b1c42a8f15]
I20250623 08:04:43.206708 2080 raft_consensus.cc:491] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.214905 2080 raft_consensus.cc:513] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.215365 343 ts_tablet_manager.cc:1428] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20250623 08:04:43.215694 2080 leader_election.cc:290] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.216706 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]
I20250623 08:04:43.217841 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74da35cd34c14284af004001b25aa531" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.218081 32515 raft_consensus.cc:2466] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:43.218566 32589 leader_election.cc:304] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.218855 2080 raft_consensus.cc:2802] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.218962 2080 raft_consensus.cc:491] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.219059 2080 raft_consensus.cc:3058] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.219499 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_98f40c99d9ff4819b958cd84cc5dc3a9 [id=13cf327a200c490992163a70f5a275cd]
I20250623 08:04:43.220760 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1802c9f70d0f45868e11e3390504fade [id=15591f6f911f4a91a1dbbef45354fe1c]
I20250623 08:04:43.221750 2080 raft_consensus.cc:513] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.222517 2080 leader_election.cc:290] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.223485 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74da35cd34c14284af004001b25aa531" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:43.223642 32515 raft_consensus.cc:3058] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.226347 32515 raft_consensus.cc:2466] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:43.227401 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 262) took 5767 ms. Trace:
I20250623 08:04:43.227483 32589 leader_election.cc:304] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.227492 952 rpcz_store.cc:276] 0623 08:04:37.459596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.489742 (+ 30146us) service_pool.cc:225] Handling call
0623 08:04:43.114082 (+5624340us) ts_tablet_manager.cc:1145] Deleting tablet 771ecfc41e2b4007a4d213cec5b128c6
0623 08:04:43.114092 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.192807 (+ 78715us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.213096 (+ 20289us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.227382 (+ 14286us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1792,"tablet-delete.queue_time_us":5624151}
I20250623 08:04:43.228560 343 tablet_bootstrap.cc:492] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.228921 952 tablet_replica.cc:331] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.229557 952 raft_consensus.cc:2241] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.230166 2013 raft_consensus.cc:2802] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.230203 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_bc37bf6911b5401c904eef106dde3592 [id=19ae0916b8c6480f93b92da98de7ec4d]
I20250623 08:04:43.230280 343 tablet_bootstrap.cc:654] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.230312 2013 raft_consensus.cc:695] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:43.230602 2013 consensus_queue.cc:237] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.230974 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_f7811b4c393d498cbf95abe8a8cdcd38 [id=1c335fc6495f4256a2d7e1f4fd66e875]
I20250623 08:04:43.231385 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_b523bec7182443139e891357d295e3a1 [id=1c4298a5f9014d8eb4ccad8c8fbdeeac]
I20250623 08:04:43.232298 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5]
I20250623 08:04:43.233351 342 tablet_bootstrap.cc:492] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.234826 342 ts_tablet_manager.cc:1397] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.003s
I20250623 08:04:43.235707 342 raft_consensus.cc:357] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.235898 342 raft_consensus.cc:383] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.235967 342 raft_consensus.cc:738] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.236256 342 consensus_queue.cc:260] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.237010 342 ts_tablet_manager.cc:1428] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:43.237362 342 tablet_bootstrap.cc:492] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.239262 342 tablet_bootstrap.cc:654] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.216183 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74da35cd34c14284af004001b25aa531" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.219435 339 tablet_bootstrap.cc:492] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.223240 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74da35cd34c14284af004001b25aa531" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:43.239842 32589 leader_election.cc:343] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.240126 339 ts_tablet_manager.cc:1397] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.003s
W20250623 08:04:43.240226 32589 leader_election.cc:343] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.240350 952 raft_consensus.cc:2270] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.240872 952 ts_tablet_manager.cc:1905] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.240856 339 raft_consensus.cc:357] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.241029 339 raft_consensus.cc:383] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.241089 339 raft_consensus.cc:738] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.241312 339 consensus_queue.cc:260] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.241895 339 ts_tablet_manager.cc:1428] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:43.242170 339 tablet_bootstrap.cc:492] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.242254 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3]
I20250623 08:04:43.246044 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_43ff9ceffb4042b5ab1b041b76bdfc96 [id=2013d4281dc84c089dc17304aefd0837]
I20250623 08:04:43.246336 981 log.cc:1199] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/ecb422550efd40cf82bc92fca5c25eda
I20250623 08:04:43.246969 1359 raft_consensus.cc:491] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.247104 1359 raft_consensus.cc:513] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.247493 981 ts_tablet_manager.cc:1939] T ecb422550efd40cf82bc92fca5c25eda P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.247681 1359 leader_election.cc:290] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.248137 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "344ebe1a14bc4d3389b0161ff102f05c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.248245 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 273) took 6358 ms. Trace:
I20250623 08:04:43.248333 981 rpcz_store.cc:276] 0623 08:04:36.889907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.890189 (+ 282us) service_pool.cc:225] Handling call
0623 08:04:43.093367 (+6203178us) ts_tablet_manager.cc:1145] Deleting tablet ecb422550efd40cf82bc92fca5c25eda
0623 08:04:43.134821 (+ 41454us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.197245 (+ 62424us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.200429 (+ 3184us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.248226 (+ 47797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6201914}
I20250623 08:04:43.248721 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 276) took 6348 ms. Trace:
W20250623 08:04:43.248703 32453 leader_election.cc:343] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.248803 981 rpcz_store.cc:276] 0623 08:04:36.900180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.900260 (+ 80us) service_pool.cc:225] Handling call
0623 08:04:43.248623 (+6348363us) ts_tablet_manager.cc:1145] Deleting tablet e22f9bae5f7248af99cec06ce6dbbd41
0623 08:04:43.248632 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.248704 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6346223}
I20250623 08:04:43.249086 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 275) took 6358 ms. Trace:
I20250623 08:04:43.249321 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "344ebe1a14bc4d3389b0161ff102f05c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.249712 32453 leader_election.cc:343] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.249817 32453 leader_election.cc:304] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.250056 1359 raft_consensus.cc:2747] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.250329 339 tablet_bootstrap.cc:654] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.252993 1359 raft_consensus.cc:491] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.253119 1359 raft_consensus.cc:513] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.253649 1359 leader_election.cc:290] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.254159 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d19d468ce13d4108a9badc36fc503306" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.254417 32646 raft_consensus.cc:2466] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.255151 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d19d468ce13d4108a9badc36fc503306" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.255177 32453 leader_election.cc:304] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.255391 1359 raft_consensus.cc:2802] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.255473 1359 raft_consensus.cc:491] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.255535 1359 raft_consensus.cc:3058] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:43.255614 32453 leader_election.cc:343] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.258163 1359 raft_consensus.cc:513] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.258661 979 ts_tablet_manager.cc:1918] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.258697 1359 leader_election.cc:290] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.258818 979 log.cc:1199] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/4c9d1c8a67e149478a0d69842a4eb5b2
I20250623 08:04:43.259191 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d19d468ce13d4108a9badc36fc503306" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.259408 32646 raft_consensus.cc:3058] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.262348 32646 raft_consensus.cc:2466] T d19d468ce13d4108a9badc36fc503306 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.263056 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d19d468ce13d4108a9badc36fc503306" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.263253 32453 leader_election.cc:304] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.263613 1359 raft_consensus.cc:2802] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
W20250623 08:04:43.263738 32453 leader_election.cc:343] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.263743 1359 raft_consensus.cc:695] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.249161 981 rpcz_store.cc:276] 0623 08:04:36.890508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:36.891815 (+ 1307us) service_pool.cc:225] Handling call
0623 08:04:43.249004 (+6357189us) ts_tablet_manager.cc:1145] Deleting tablet d0ee5a4fe03848229524cc0105226d8c
0623 08:04:43.249013 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.249073 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6333699}
I20250623 08:04:43.264130 981 tablet_replica.cc:331] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.264158 1359 consensus_queue.cc:237] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.264320 981 raft_consensus.cc:2241] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.264396 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_738cccce8e6c47d683b9a7c569aa58e3 [id=20baa502b37c4ec1904f52ddeaf99c34]
I20250623 08:04:43.264492 981 raft_consensus.cc:2270] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.265038 981 ts_tablet_manager.cc:1905] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.265141 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]
I20250623 08:04:43.265606 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_4d2fc00363204f8485905af018f7911a [id=22f134d5d55f40ef98dec2f14a6d55ce]
I20250623 08:04:43.266593 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1a1b8de8d0a645e9a48938956c562be9 [id=2562e6968e504a5aa52358337d79f528]
I20250623 08:04:43.269047 342 tablet_bootstrap.cc:492] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.269192 342 ts_tablet_manager.cc:1397] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250623 08:04:43.269846 342 raft_consensus.cc:357] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.270012 342 raft_consensus.cc:383] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.270069 342 raft_consensus.cc:738] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.270258 342 consensus_queue.cc:260] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.270743 342 ts_tablet_manager.cc:1428] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:43.271047 343 tablet_bootstrap.cc:492] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.271080 342 tablet_bootstrap.cc:492] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.271209 343 ts_tablet_manager.cc:1397] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.005s
I20250623 08:04:43.272010 343 raft_consensus.cc:357] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.272194 343 raft_consensus.cc:383] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.272276 343 raft_consensus.cc:738] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.272565 342 tablet_bootstrap.cc:654] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.274386 339 tablet_bootstrap.cc:492] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.274524 339 ts_tablet_manager.cc:1397] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250623 08:04:43.275326 339 raft_consensus.cc:357] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.275499 339 raft_consensus.cc:383] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.275554 339 raft_consensus.cc:738] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.276077 343 consensus_queue.cc:260] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.277362 343 ts_tablet_manager.cc:1428] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:43.278144 343 tablet_bootstrap.cc:492] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.279300 981 ts_tablet_manager.cc:1918] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:43.279493 981 log.cc:1199] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/136e482995a24c5daec55a701f3091fa
I20250623 08:04:43.279798 343 tablet_bootstrap.cc:654] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.280257 981 ts_tablet_manager.cc:1939] T 136e482995a24c5daec55a701f3091fa P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.281124 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 278) took 6280 ms. Trace:
I20250623 08:04:43.281271 981 rpcz_store.cc:276] 0623 08:04:37.000494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.000545 (+ 51us) service_pool.cc:225] Handling call
0623 08:04:43.264083 (+6263538us) ts_tablet_manager.cc:1145] Deleting tablet 136e482995a24c5daec55a701f3091fa
0623 08:04:43.264093 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.275439 (+ 11346us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.279201 (+ 3762us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.281104 (+ 1903us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6263354}
I20250623 08:04:43.281697 981 tablet_replica.cc:331] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.281914 981 raft_consensus.cc:2241] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.282112 981 raft_consensus.cc:2270] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.283061 981 ts_tablet_manager.cc:1905] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.284332 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]
I20250623 08:04:43.285423 339 consensus_queue.cc:260] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.285921 339 ts_tablet_manager.cc:1428] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250623 08:04:43.259186 979 ts_tablet_manager.cc:1939] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:43.286214 339 tablet_bootstrap.cc:492] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.287112 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 390) took 2050 ms. Trace:
I20250623 08:04:43.287842 339 tablet_bootstrap.cc:654] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.289228 952 ts_tablet_manager.cc:1918] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:43.289466 952 log.cc:1199] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/c0773fc60d4a463596f536f15d58bf57
I20250623 08:04:43.289845 952 ts_tablet_manager.cc:1939] T c0773fc60d4a463596f536f15d58bf57 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.290849 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 276) took 5492 ms. Trace:
I20250623 08:04:43.292171 952 rpcz_store.cc:276] 0623 08:04:37.798162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.798443 (+ 281us) service_pool.cc:225] Handling call
0623 08:04:43.228868 (+5430425us) ts_tablet_manager.cc:1145] Deleting tablet c0773fc60d4a463596f536f15d58bf57
0623 08:04:43.228880 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.244231 (+ 15351us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.289114 (+ 44883us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.290829 (+ 1715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5430223}
I20250623 08:04:43.292110 342 tablet_bootstrap.cc:492] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.292649 342 ts_tablet_manager.cc:1397] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250623 08:04:43.293395 342 raft_consensus.cc:357] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.293622 342 raft_consensus.cc:383] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.293722 342 raft_consensus.cc:738] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.293980 342 consensus_queue.cc:260] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.294502 342 ts_tablet_manager.cc:1428] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:43.294935 342 tablet_bootstrap.cc:492] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.304355 342 tablet_bootstrap.cc:654] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.304642 952 tablet_replica.cc:331] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.307080 952 raft_consensus.cc:2241] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.307360 952 pending_rounds.cc:70] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:43.307468 952 pending_rounds.cc:77] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727456156246016 op_type: NO_OP noop_request { }
I20250623 08:04:43.307602 339 tablet_bootstrap.cc:492] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.307649 952 raft_consensus.cc:2887] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:43.307754 339 ts_tablet_manager.cc:1397] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20250623 08:04:43.307803 981 ts_tablet_manager.cc:1918] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250623 08:04:43.307888 952 raft_consensus.cc:2270] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.307987 981 log.cc:1199] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/283bebc7063d437ba3e541d7dee3c0b5
I20250623 08:04:43.308455 981 ts_tablet_manager.cc:1939] T 283bebc7063d437ba3e541d7dee3c0b5 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.308493 339 raft_consensus.cc:357] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.308676 339 raft_consensus.cc:383] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.308722 952 ts_tablet_manager.cc:1905] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.308733 339 raft_consensus.cc:738] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.309044 339 consensus_queue.cc:260] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.309932 339 ts_tablet_manager.cc:1428] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:43.310196 339 tablet_bootstrap.cc:492] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.311879 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_bd4ee5f643fe4d9bab6b2f11ef0869cf [id=28106e3ea0bd442e85bbbe3e26f8f0d4]
I20250623 08:04:43.312119 339 tablet_bootstrap.cc:654] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.312642 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_4a19c65595654fa98efc42626b6a9d85 [id=2979e4296458439ea5753b6ba67634ac]
I20250623 08:04:43.313120 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_8ec723069381477ca64f5aaad0df76fb [id=2a5a01b05a554ac1ad6abd21ad00d734]
I20250623 08:04:43.313573 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_9aeb558e2ae64fe2b45bef5d2b6a2625 [id=2bfacaaab9f347b1a77e5eb29a9c485e]
I20250623 08:04:43.314046 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_306d24f2f3484139b3586e0a1ebb8562 [id=2f72a0d782b54f109937d1ab340afd89]
I20250623 08:04:43.314515 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]
I20250623 08:04:43.318897 343 tablet_bootstrap.cc:492] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.319106 343 ts_tablet_manager.cc:1397] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.001s
I20250623 08:04:43.319902 343 raft_consensus.cc:357] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.320122 343 raft_consensus.cc:383] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.320209 343 raft_consensus.cc:738] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.320485 343 consensus_queue.cc:260] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.321099 343 ts_tablet_manager.cc:1428] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:43.321867 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 279) took 6321 ms. Trace:
I20250623 08:04:43.331676 343 tablet_bootstrap.cc:492] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.333626 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_0e43b0bc6b7b47b1b59882856aaa9736 [id=30949c1b60bc4300852bcaf533ceed4c]
I20250623 08:04:43.333685 343 tablet_bootstrap.cc:654] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.333846 981 rpcz_store.cc:276] 0623 08:04:37.000599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.000759 (+ 160us) service_pool.cc:225] Handling call
0623 08:04:43.281651 (+6280892us) ts_tablet_manager.cc:1145] Deleting tablet 283bebc7063d437ba3e541d7dee3c0b5
0623 08:04:43.281661 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.302792 (+ 21131us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.307696 (+ 4904us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.321844 (+ 14148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6280735}
I20250623 08:04:43.334123 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524]
I20250623 08:04:43.334363 981 tablet_replica.cc:331] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.334560 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9]
I20250623 08:04:43.335426 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e]
I20250623 08:04:43.335713 981 raft_consensus.cc:2241] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.335988 981 raft_consensus.cc:2270] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.336623 981 ts_tablet_manager.cc:1905] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.336614 1957 raft_consensus.cc:491] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.336764 1957 raft_consensus.cc:513] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.337356 1957 leader_election.cc:290] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.337908 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16331ba6a34942a5a1b4ff4ac7bac55e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.338184 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16331ba6a34942a5a1b4ff4ac7bac55e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.338379 32646 raft_consensus.cc:2466] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.338907 32453 leader_election.cc:304] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:43.339167 32453 leader_election.cc:343] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.339433 1957 raft_consensus.cc:2802] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.339514 1957 raft_consensus.cc:491] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.339586 1957 raft_consensus.cc:3058] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.342206 1957 raft_consensus.cc:513] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.342898 1957 leader_election.cc:290] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.343664 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16331ba6a34942a5a1b4ff4ac7bac55e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.343847 32646 raft_consensus.cc:3058] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.345124 343 tablet_bootstrap.cc:492] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.345300 343 ts_tablet_manager.cc:1397] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20250623 08:04:43.346047 343 raft_consensus.cc:357] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.346257 343 raft_consensus.cc:383] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.346351 343 raft_consensus.cc:738] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.346532 32646 raft_consensus.cc:2466] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.346714 343 consensus_queue.cc:260] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.347035 1843 consensus_queue.cc:1035] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.336107 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9]
I20250623 08:04:43.347440 32453 leader_election.cc:304] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.347554 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16331ba6a34942a5a1b4ff4ac7bac55e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.347723 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]
I20250623 08:04:43.348062 1359 raft_consensus.cc:2802] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.348134 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_b9220275c4064c11bb0618d6a459aab6 [id=392754cfbca449d887b33eb6ec5f0ae2]
I20250623 08:04:43.348187 1359 raft_consensus.cc:695] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.348500 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_e14bd6f5e20e42b1898bce238d4d986c [id=3a0c95fd4bd84b80ad1f5951cbef92e8]
I20250623 08:04:43.348417 1359 consensus_queue.cc:237] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:43.349157 32453 leader_election.cc:343] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.350701 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_2ecc70ddf2dc40e081fefb676d46a68d [id=3a1a4d6d2cc24dc8abe85a4597c0d0c8]
I20250623 08:04:43.351148 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_c748f600d56f4374bdb31ad502ec7557 [id=3b342fcf5bff4bf0bb19c90fca9015c3]
I20250623 08:04:43.351116 979 rpcz_store.cc:276] 0623 08:04:41.237077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.237139 (+ 62us) service_pool.cc:225] Handling call
0623 08:04:43.159726 (+1922587us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d1c8a67e149478a0d69842a4eb5b2
0623 08:04:43.159736 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.212542 (+ 52806us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.258516 (+ 45974us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.287073 (+ 28557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1922449}
I20250623 08:04:43.351598 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]
I20250623 08:04:43.351955 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 391) took 2041 ms. Trace:
I20250623 08:04:43.352030 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_8ab4ba6029964c2fa123202d6c5c836d [id=3c1572742d694d48be03ccabe9166f8d]
I20250623 08:04:43.352136 979 rpcz_store.cc:276] 0623 08:04:41.310173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.310233 (+ 60us) service_pool.cc:225] Handling call
0623 08:04:43.351807 (+2041574us) ts_tablet_manager.cc:1145] Deleting tablet 3136ec837bea4619ae3995ceec9b24d6
0623 08:04:43.351822 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.351938 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2041377}
I20250623 08:04:43.352535 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]
I20250623 08:04:43.352774 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 394) took 1385 ms. Trace:
I20250623 08:04:43.352880 979 rpcz_store.cc:276] 0623 08:04:41.967447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.967518 (+ 71us) service_pool.cc:225] Handling call
0623 08:04:43.352646 (+1385128us) ts_tablet_manager.cc:1145] Deleting tablet ba5913811e9343389c9ea0ec8b8c05ef
0623 08:04:43.352670 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.352760 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1374719}
I20250623 08:04:43.353487 979 tablet_replica.cc:331] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:43.353729 979 raft_consensus.cc:2241] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.353917 979 raft_consensus.cc:2270] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.354768 343 ts_tablet_manager.cc:1428] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250623 08:04:43.355917 343 tablet_bootstrap.cc:492] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.360863 343 tablet_bootstrap.cc:654] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.361109 339 tablet_bootstrap.cc:492] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.361256 339 ts_tablet_manager.cc:1397] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20250623 08:04:43.361979 339 raft_consensus.cc:357] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.362149 339 raft_consensus.cc:383] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.362200 339 raft_consensus.cc:738] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.362390 339 consensus_queue.cc:260] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.362995 339 ts_tablet_manager.cc:1428] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:43.364296 339 tablet_bootstrap.cc:492] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.366183 979 ts_tablet_manager.cc:1905] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.366291 339 tablet_bootstrap.cc:654] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.366930 1843 raft_consensus.cc:491] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.367080 1843 raft_consensus.cc:513] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:43.367619 32020 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-taskJxinPB/build/asan/bin/kudu pid 32028: Timed out: CheckLeaks RPC to 127.31.69.62:39465 timed out after 1.000s (SENT)
I20250623 08:04:43.367702 1843 leader_election.cc:290] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.367799 32020 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskJxinPB/build/asan/bin/kudu with pid 32028
I20250623 08:04:43.369663 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8a5e35c1974456b8079ab8907c2652e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.371059 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8a5e35c1974456b8079ab8907c2652e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.371496 32314 leader_election.cc:304] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.373308 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_8e6f4c63178440f4ba5c9f65d213df3d [id=3d71395c0fab47b68b1dcc47bfdc46fb]
I20250623 08:04:43.374823 1910 raft_consensus.cc:491] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.374954 1910 raft_consensus.cc:513] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.375108 952 ts_tablet_manager.cc:1918] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.375203 952 log.cc:1199] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f9f4829e103b460eb582beff3b966bb4
I20250623 08:04:43.375532 952 ts_tablet_manager.cc:1939] T f9f4829e103b460eb582beff3b966bb4 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.375576 1910 leader_election.cc:290] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.375994 1910 raft_consensus.cc:2747] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.376327 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 277) took 5577 ms. Trace:
I20250623 08:04:43.376530 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]
I20250623 08:04:43.376430 952 rpcz_store.cc:276] 0623 08:04:37.799721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.799869 (+ 148us) service_pool.cc:225] Handling call
0623 08:04:43.304593 (+5504724us) ts_tablet_manager.cc:1145] Deleting tablet f9f4829e103b460eb582beff3b966bb4
0623 08:04:43.304604 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.314723 (+ 10119us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.374995 (+ 60272us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.376307 (+ 1312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5504549}
I20250623 08:04:43.376982 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_6b3d1637c4c1443089f04128e190256f [id=3f48141767f441dbbdf5a7742a16176f]
I20250623 08:04:43.378393 1682 raft_consensus.cc:491] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.378500 1682 raft_consensus.cc:513] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.378818 342 tablet_bootstrap.cc:492] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.378989 342 ts_tablet_manager.cc:1397] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.084s user 0.005s sys 0.002s
I20250623 08:04:43.379107 1682 leader_election.cc:290] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.379244 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_712680c326284dd782d57dc0ee153879 [id=407f4156ebee45ecb729ad3b3d5f26e5]
I20250623 08:04:43.379804 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8945b6f49474648ac73bcb85285680a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.379772 342 raft_consensus.cc:357] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.379952 342 raft_consensus.cc:383] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.380033 32651 raft_consensus.cc:2391] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:43.380026 342 raft_consensus.cc:738] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.380221 342 consensus_queue.cc:260] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.381042 342 ts_tablet_manager.cc:1428] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:43.381340 342 tablet_bootstrap.cc:492] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.382575 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1be8f8557b9a4ae2917c37b8d01cedf6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.382795 32651 raft_consensus.cc:2391] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:43.383148 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_56c6dd0a0ee64ed088889c23db0fd2e9 [id=4199d682d171458ab79e4a36807c7f6b]
I20250623 08:04:43.383459 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1be8f8557b9a4ae2917c37b8d01cedf6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:43.386588 32314 consensus_peers.cc:487] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c01121f2e644d6aa6e8ec228bf75be9. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:43.387070 1682 consensus_queue.cc:1035] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.387423 952 tablet_replica.cc:331] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.387638 952 raft_consensus.cc:2241] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.387753 32314 leader_election.cc:304] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.387909 952 pending_rounds.cc:70] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:43.388003 1682 raft_consensus.cc:3058] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.388001 952 pending_rounds.cc:77] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727456527667200 op_type: NO_OP noop_request { }
I20250623 08:04:43.388096 952 raft_consensus.cc:2887] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:43.388188 952 raft_consensus.cc:2270] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.391278 1682 raft_consensus.cc:2747] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.391476 952 ts_tablet_manager.cc:1905] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.392247 339 tablet_bootstrap.cc:492] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.392369 339 ts_tablet_manager.cc:1397] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.001s
I20250623 08:04:43.393110 339 raft_consensus.cc:357] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.393254 339 raft_consensus.cc:383] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.393301 339 raft_consensus.cc:738] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.393476 339 consensus_queue.cc:260] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.393894 339 ts_tablet_manager.cc:1428] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:43.394394 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8945b6f49474648ac73bcb85285680a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.394587 339 tablet_bootstrap.cc:492] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.394979 2013 raft_consensus.cc:491] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.395098 2013 raft_consensus.cc:513] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.395745 2013 leader_election.cc:290] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.396306 339 tablet_bootstrap.cc:654] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.396409 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f217a8eb12014fcf89b9df4a2cddf5f6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.397110 32589 leader_election.cc:343] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.395015 342 tablet_bootstrap.cc:654] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.396405 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f217a8eb12014fcf89b9df4a2cddf5f6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.399566 32518 raft_consensus.cc:2466] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:43.399948 32589 leader_election.cc:304] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.400367 2013 raft_consensus.cc:2802] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.400440 2013 raft_consensus.cc:491] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.400501 2013 raft_consensus.cc:3058] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.402772 1957 raft_consensus.cc:491] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.402926 1957 raft_consensus.cc:513] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.403467 1957 leader_election.cc:290] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.404665 2013 raft_consensus.cc:513] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.405520 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f217a8eb12014fcf89b9df4a2cddf5f6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:43.406000 32589 leader_election.cc:343] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.406229 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb06a340f82047ac8ec3516edb08f4b6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.406369 32651 raft_consensus.cc:2466] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.406797 32453 leader_election.cc:304] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.407290 1957 raft_consensus.cc:2802] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.407387 1957 raft_consensus.cc:491] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.407467 1957 raft_consensus.cc:3058] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.409085 979 ts_tablet_manager.cc:1918] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.409188 979 log.cc:1199] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/adc3cf4355c24b7395fcb22af8520346
I20250623 08:04:43.409507 979 ts_tablet_manager.cc:1939] T adc3cf4355c24b7395fcb22af8520346 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:43.410125 1957 raft_consensus.cc:513] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.410274 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 395) took 1440 ms. Trace:
I20250623 08:04:43.410343 979 rpcz_store.cc:276] 0623 08:04:41.969370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.969415 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:43.353438 (+1384023us) ts_tablet_manager.cc:1145] Deleting tablet adc3cf4355c24b7395fcb22af8520346
0623 08:04:43.353447 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.389851 (+ 36404us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.408990 (+ 19139us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.410256 (+ 1266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1374466}
I20250623 08:04:43.410701 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]
I20250623 08:04:43.411340 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_9c6bc2fbfc274d2e8414205f629cb439 [id=42f65c762e2b46469d3ff189b698b447]
I20250623 08:04:43.411681 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb06a340f82047ac8ec3516edb08f4b6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.411826 32654 raft_consensus.cc:3058] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.412079 1359 raft_consensus.cc:491] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.412205 1359 raft_consensus.cc:513] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.412451 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_c4c658393f424eb1ac22133856d0d0b1 [id=44217c9c758e4530aa497a405168eac0]
I20250623 08:04:43.412741 1359 leader_election.cc:290] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.412822 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]
I20250623 08:04:43.413854 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_3980cd736bd342f4bcd5cf289c1c8c25 [id=47a9c2f4140743c3a357032721181f32]
I20250623 08:04:43.415120 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]
I20250623 08:04:43.417500 2013 leader_election.cc:290] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.417533 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3136ec837bea4619ae3995ceec9b24d6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.418426 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f217a8eb12014fcf89b9df4a2cddf5f6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:43.418645 32518 raft_consensus.cc:3058] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:43.418426 32453 leader_election.cc:343] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.418962 32654 raft_consensus.cc:2466] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.419356 32453 leader_election.cc:304] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.410688 1957 leader_election.cc:290] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.419690 1957 raft_consensus.cc:2802] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.419907 1957 raft_consensus.cc:695] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.420198 1957 consensus_queue.cc:237] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.418962 979 tablet_replica.cc:331] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:43.421500 32518 raft_consensus.cc:2466] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:43.421663 979 raft_consensus.cc:2241] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.421798 979 raft_consensus.cc:2270] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.421922 32589 leader_election.cc:304] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.422250 2013 raft_consensus.cc:2802] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.422631 2013 raft_consensus.cc:695] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:43.423007 2013 consensus_queue.cc:237] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.423309 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_47c06fdd0628467d8a0727b9adeb890d [id=4aea4a71b2e64e81bf9cca2a06f15521]
I20250623 08:04:43.425321 979 ts_tablet_manager.cc:1905] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.427289 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_787415ae5937475f90f90000aa575d0e [id=4b27489168854629848531915104847a]
I20250623 08:04:43.429602 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb06a340f82047ac8ec3516edb08f4b6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.429929 981 ts_tablet_manager.cc:1918] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:43.429957 952 ts_tablet_manager.cc:1918] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.430066 952 log.cc:1199] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/2c5ecd4fa5574a1b90b6e82f89ee88f7
I20250623 08:04:43.430306 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb06a340f82047ac8ec3516edb08f4b6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.430723 32314 leader_election.cc:304] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.430756 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3136ec837bea4619ae3995ceec9b24d6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.432453 32453 leader_election.cc:343] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.432502 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_ac62bae678de4b11a971f7bbccd50303 [id=4f4663a5cad14a3593e16a52ec8aeece]
I20250623 08:04:43.432811 1682 raft_consensus.cc:3058] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.432931 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_b68f23253200457683bb5cedcc56041f [id=4f81957f51bd40c192ad7c716cbd893f]
I20250623 08:04:43.432982 952 ts_tablet_manager.cc:1939] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.433671 343 tablet_bootstrap.cc:492] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.433671 1843 raft_consensus.cc:491] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.433779 1843 raft_consensus.cc:513] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:43.433938 343 ts_tablet_manager.cc:1397] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.004s
I20250623 08:04:43.434265 1843 leader_election.cc:290] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.434926 343 raft_consensus.cc:357] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.435159 343 raft_consensus.cc:383] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.435256 343 raft_consensus.cc:738] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.435537 343 consensus_queue.cc:260] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.436162 343 ts_tablet_manager.cc:1428] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:43.436987 1682 raft_consensus.cc:2747] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.439580 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 278) took 5633 ms. Trace:
I20250623 08:04:43.439965 343 tablet_bootstrap.cc:492] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.441665 343 tablet_bootstrap.cc:654] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.441812 952 rpcz_store.cc:276] 0623 08:04:37.806107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.806442 (+ 335us) service_pool.cc:225] Handling call
0623 08:04:43.387352 (+5580910us) ts_tablet_manager.cc:1145] Deleting tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7
0623 08:04:43.387364 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.407093 (+ 19729us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.429828 (+ 22735us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.439539 (+ 9711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5580718}
I20250623 08:04:43.442327 952 tablet_replica.cc:331] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.442637 952 raft_consensus.cc:2241] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.442914 952 raft_consensus.cc:2270] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.444608 342 tablet_bootstrap.cc:492] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.444810 342 ts_tablet_manager.cc:1397] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.004s
I20250623 08:04:43.445595 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_e8864c4ee64b4f4dbf04caa321024840 [id=537cece72a774ac99261a4c2a2e52c2e]
I20250623 08:04:43.445639 342 raft_consensus.cc:357] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.445951 342 raft_consensus.cc:383] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.446087 342 raft_consensus.cc:738] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.446369 342 consensus_queue.cc:260] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.447129 342 ts_tablet_manager.cc:1428] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:43.448300 342 tablet_bootstrap.cc:492] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.451048 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_5d6ec4f6805b4e0883b8d0419625d033 [id=545b478879aa4f4cb35393e3065d3ff7]
W20250623 08:04:43.449915 32453 leader_election.cc:343] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.450709 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c9d1c8a67e149478a0d69842a4eb5b2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:43.455590 32453 leader_election.cc:343] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.455739 32453 leader_election.cc:304] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.455984 1359 raft_consensus.cc:2747] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.456681 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22]
I20250623 08:04:43.447739 952 ts_tablet_manager.cc:1905] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.458628 979 ts_tablet_manager.cc:1918] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:43.458741 979 log.cc:1199] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/1be8f8557b9a4ae2917c37b8d01cedf6
I20250623 08:04:43.461316 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_79fce459ec304a14823e0a2179f6b804 [id=57b3f76fe53a4f69bfb544ad72a7ca52]
I20250623 08:04:43.431190 981 log.cc:1199] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/771ecfc41e2b4007a4d213cec5b128c6
I20250623 08:04:43.466267 981 ts_tablet_manager.cc:1939] T 771ecfc41e2b4007a4d213cec5b128c6 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.466312 952 ts_tablet_manager.cc:1918] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.466423 952 log.cc:1199] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f83f68ebda7e4826bbb17278cd4ac5a8
W20250623 08:04:43.461824 1213 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.69.62:39465 timed out after 0.514s (SENT)
I20250623 08:04:43.467430 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 280) took 6466 ms. Trace:
I20250623 08:04:43.467530 981 rpcz_store.cc:276] 0623 08:04:37.000708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.000945 (+ 237us) service_pool.cc:225] Handling call
0623 08:04:43.334310 (+6333365us) ts_tablet_manager.cc:1145] Deleting tablet 771ecfc41e2b4007a4d213cec5b128c6
0623 08:04:43.334322 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.341112 (+ 6790us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.429828 (+ 88716us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.467410 (+ 37582us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6333248}
I20250623 08:04:43.467880 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 281) took 6467 ms. Trace:
I20250623 08:04:43.467940 981 rpcz_store.cc:276] 0623 08:04:37.000799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.001080 (+ 281us) service_pool.cc:225] Handling call
0623 08:04:43.467759 (+6466679us) ts_tablet_manager.cc:1145] Deleting tablet a34da529ffde4595a23fc6596b30e3e9
0623 08:04:43.467769 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.467867 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6466566}
I20250623 08:04:43.468140 981 tablet_replica.cc:331] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.468289 981 raft_consensus.cc:2241] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.468439 981 raft_consensus.cc:2270] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.468998 981 ts_tablet_manager.cc:1905] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:43.457830 32314 leader_election.cc:343] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 4c9d1c8a67e149478a0d69842a4eb5b2
I20250623 08:04:43.469602 32314 leader_election.cc:304] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.469741 342 tablet_bootstrap.cc:654] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.470702 1682 raft_consensus.cc:2747] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.474442 952 ts_tablet_manager.cc:1939] T f83f68ebda7e4826bbb17278cd4ac5a8 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.475261 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 279) took 5668 ms. Trace:
I20250623 08:04:43.476265 32368 raft_consensus.cc:3058] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.476440 952 rpcz_store.cc:276] 0623 08:04:37.807458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.807784 (+ 326us) service_pool.cc:225] Handling call
0623 08:04:43.442252 (+5634468us) ts_tablet_manager.cc:1145] Deleting tablet f83f68ebda7e4826bbb17278cd4ac5a8
0623 08:04:43.442264 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.461296 (+ 19032us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.466189 (+ 4893us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.475242 (+ 9053us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5634310}
I20250623 08:04:43.477193 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 280) took 5668 ms. Trace:
I20250623 08:04:43.477617 952 rpcz_store.cc:276] 0623 08:04:37.808682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.809099 (+ 417us) service_pool.cc:225] Handling call
0623 08:04:43.477078 (+5667979us) ts_tablet_manager.cc:1145] Deleting tablet cde7df7d39a54114a4dcb28cfd1eb429
0623 08:04:43.477088 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.477154 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5667835}
I20250623 08:04:43.479000 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_e216bbb5ac2a4720b36183e30bd6ed91 [id=57e548bc818e4968b432cc3af298e675]
I20250623 08:04:43.462859 343 tablet_bootstrap.cc:492] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.479318 343 ts_tablet_manager.cc:1397] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20250623 08:04:43.479566 979 ts_tablet_manager.cc:1939] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:43.480052 343 raft_consensus.cc:357] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.480198 343 raft_consensus.cc:383] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.480242 343 raft_consensus.cc:738] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.480345 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 407) took 1221 ms. Trace:
I20250623 08:04:43.480423 343 consensus_queue.cc:260] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.480449 979 rpcz_store.cc:276] 0623 08:04:42.259130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.285033 (+ 25903us) service_pool.cc:225] Handling call
0623 08:04:43.418891 (+1133858us) ts_tablet_manager.cc:1145] Deleting tablet 1be8f8557b9a4ae2917c37b8d01cedf6
0623 08:04:43.418920 (+ 29us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.435388 (+ 16468us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.458522 (+ 23134us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.480327 (+ 21805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133522}
I20250623 08:04:43.480917 343 ts_tablet_manager.cc:1428] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:43.481154 343 tablet_bootstrap.cc:492] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.481349 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_a935cce17e794f328640dde460d22086 [id=5847481518254b318e71e8198962267a]
I20250623 08:04:43.482060 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_d87c4052e62d4c1c87fc25cff314319b [id=58d5dec0c723470a9a7e56ce37dc71d0]
I20250623 08:04:43.482498 343 tablet_bootstrap.cc:654] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.488538 979 tablet_replica.cc:331] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:43.488804 979 raft_consensus.cc:2241] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.489056 979 pending_rounds.cc:70] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:43.489588 979 pending_rounds.cc:77] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727455425306624 op_type: NO_OP noop_request { }
I20250623 08:04:43.490311 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 281) took 5680 ms. Trace:
I20250623 08:04:43.490453 979 raft_consensus.cc:2887] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:43.482805 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873]
I20250623 08:04:43.490536 952 rpcz_store.cc:276] 0623 08:04:37.809989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.810253 (+ 264us) service_pool.cc:225] Handling call
0623 08:04:43.490165 (+5679912us) ts_tablet_manager.cc:1145] Deleting tablet f0117cbb09dd4067836266c9d6a32adf
0623 08:04:43.490174 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.490262 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5679753}
I20250623 08:04:43.491123 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 282) took 5680 ms. Trace:
I20250623 08:04:43.491201 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_5cfbfaf89e6a4bdf94cef7019cf9c5e4 [id=5c35ac21e1964919acdfdc5e72e5b6b7]
I20250623 08:04:43.491317 952 rpcz_store.cc:276] 0623 08:04:37.811217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.811551 (+ 334us) service_pool.cc:225] Handling call
0623 08:04:43.491016 (+5679465us) ts_tablet_manager.cc:1145] Deleting tablet 9bc95f12ab5d4b9b9f0729cb88aabbaa
0623 08:04:43.491025 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.491079 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5679323}
I20250623 08:04:43.491681 979 raft_consensus.cc:2270] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.491892 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 284) took 5678 ms. Trace:
I20250623 08:04:43.492041 952 rpcz_store.cc:276] 0623 08:04:37.814100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.814151 (+ 51us) service_pool.cc:225] Handling call
0623 08:04:43.491798 (+5677647us) ts_tablet_manager.cc:1145] Deleting tablet 19536c16ff32460e9db571e4ce411bab
0623 08:04:43.491808 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.491861 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5677509}
I20250623 08:04:43.492833 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 285) took 5678 ms. Trace:
I20250623 08:04:43.493511 952 rpcz_store.cc:276] 0623 08:04:37.814936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.815127 (+ 191us) service_pool.cc:225] Handling call
0623 08:04:43.492733 (+5677606us) ts_tablet_manager.cc:1145] Deleting tablet 417a050eada4403ebbddff255fb6fb5a
0623 08:04:43.492742 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.492795 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5677460}
I20250623 08:04:43.493767 981 ts_tablet_manager.cc:1918] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.494158 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]
I20250623 08:04:43.494464 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 286) took 5678 ms. Trace:
I20250623 08:04:43.494904 2080 raft_consensus.cc:491] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.495044 2080 raft_consensus.cc:513] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.495271 952 rpcz_store.cc:276] 0623 08:04:37.816041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.816297 (+ 256us) service_pool.cc:225] Handling call
0623 08:04:43.494371 (+5678074us) ts_tablet_manager.cc:1145] Deleting tablet 5fdc9e53da264a5ca1f70e1846c906f0
0623 08:04:43.494378 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.494428 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5677917}
I20250623 08:04:43.495623 2080 leader_election.cc:290] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.496490 343 tablet_bootstrap.cc:492] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.496654 343 ts_tablet_manager.cc:1397] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20250623 08:04:43.496140 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "344ebe1a14bc4d3389b0161ff102f05c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.497244 981 log.cc:1199] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/20e7a78cfc384033810603316b9e66bf
W20250623 08:04:43.497303 32589 leader_election.cc:343] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.497392 343 raft_consensus.cc:357] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.497561 343 raft_consensus.cc:383] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.497627 343 raft_consensus.cc:738] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.497575 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "344ebe1a14bc4d3389b0161ff102f05c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.497882 32515 raft_consensus.cc:2466] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:43.497850 343 consensus_queue.cc:260] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.498284 32589 leader_election.cc:304] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.498399 343 ts_tablet_manager.cc:1428] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:43.498508 2080 raft_consensus.cc:2802] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.498576 2080 raft_consensus.cc:491] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.498641 2080 raft_consensus.cc:3058] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.498708 343 tablet_bootstrap.cc:492] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.498354 981 ts_tablet_manager.cc:1939] T 20e7a78cfc384033810603316b9e66bf P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.500506 1359 consensus_queue.cc:1035] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.500836 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 277) took 6500 ms. Trace:
I20250623 08:04:43.501158 2080 raft_consensus.cc:513] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.501685 2080 leader_election.cc:290] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.502056 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "344ebe1a14bc4d3389b0161ff102f05c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.502331 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "344ebe1a14bc4d3389b0161ff102f05c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
W20250623 08:04:43.502573 32589 leader_election.cc:343] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.502606 32515 raft_consensus.cc:3058] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.500927 981 rpcz_store.cc:276] 0623 08:04:37.000232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.003977 (+ 3745us) service_pool.cc:225] Handling call
0623 08:04:43.468107 (+6464130us) ts_tablet_manager.cc:1145] Deleting tablet 20e7a78cfc384033810603316b9e66bf
0623 08:04:43.468114 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.481550 (+ 13436us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.493634 (+ 12084us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.500816 (+ 7182us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6463958}
I20250623 08:04:43.505373 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584]
I20250623 08:04:43.505856 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 287) took 5688 ms. Trace:
I20250623 08:04:43.507313 343 tablet_bootstrap.cc:654] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.507463 952 rpcz_store.cc:276] 0623 08:04:37.817244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.817502 (+ 258us) service_pool.cc:225] Handling call
0623 08:04:43.505748 (+5688246us) ts_tablet_manager.cc:1145] Deleting tablet f3d32798168345edb9898751d083057c
0623 08:04:43.505757 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.505819 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5688106}
I20250623 08:04:43.507936 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 289) took 5682 ms. Trace:
I20250623 08:04:43.508164 952 rpcz_store.cc:276] 0623 08:04:37.850356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.850421 (+ 65us) service_pool.cc:225] Handling call
0623 08:04:43.507817 (+5657396us) ts_tablet_manager.cc:1145] Deleting tablet 966a159e212040109e2842c15b69fb71
0623 08:04:43.507827 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.507890 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5642175}
I20250623 08:04:43.508562 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 290) took 5657 ms. Trace:
W20250623 08:04:43.509195 32592 connection.cc:537] server connection from 127.0.0.1:36612 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:43.509275 385 connection.cc:537] client connection to 127.31.69.62:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:43.509420 385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250623 08:04:43.510035 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_100f074733ca4066983e68257aea0b11 [id=5d78d651cb7345fba13c371f9c1db55a]
I20250623 08:04:43.507933 32515 raft_consensus.cc:2466] T 344ebe1a14bc4d3389b0161ff102f05c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:43.510119 342 tablet_bootstrap.cc:492] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.511135 342 ts_tablet_manager.cc:1397] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.008s
I20250623 08:04:43.511632 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_de7a3df7f0064a7bb7715c0c5dcc337e [id=5e7083acd09b41b2b78bd229dc7b1e3d]
I20250623 08:04:43.512034 342 raft_consensus.cc:357] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.512267 342 raft_consensus.cc:383] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.512369 342 raft_consensus.cc:738] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.512712 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_da00c692e65345c496f4e52c106aa2e6 [id=5ea293756d514f06b8d749c615417969]
I20250623 08:04:43.512718 342 consensus_queue.cc:260] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.513480 342 ts_tablet_manager.cc:1428] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:43.513871 342 tablet_bootstrap.cc:492] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.510216 952 rpcz_store.cc:276] 0623 08:04:37.850693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.850740 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:43.508470 (+5657730us) ts_tablet_manager.cc:1145] Deleting tablet 7da84969a6464b5caeaf47af0cbaaa4d
0623 08:04:43.508479 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.508532 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5642759}
I20250623 08:04:43.514416 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 291) took 5663 ms. Trace:
I20250623 08:04:43.514534 952 rpcz_store.cc:276] 0623 08:04:37.850968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.851013 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:43.514290 (+5663277us) ts_tablet_manager.cc:1145] Deleting tablet 1e47b10f46b8402e94354ebca573c194
0623 08:04:43.514302 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.514374 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5648533}
I20250623 08:04:43.513532 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_5cf87169ed0d4987b396195d518aa055 [id=5edc67ced51e4cde9dbfcf448d24b87d]
I20250623 08:04:43.514938 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 292) took 5663 ms. Trace:
I20250623 08:04:43.515050 952 rpcz_store.cc:276] 0623 08:04:37.851277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.851331 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:43.514827 (+5663496us) ts_tablet_manager.cc:1145] Deleting tablet 175ee90133994154b7e4b58b0770db3b
0623 08:04:43.514838 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.514902 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5649030}
I20250623 08:04:43.515277 32368 raft_consensus.cc:3058] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.515332 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 293) took 5663 ms. Trace:
I20250623 08:04:43.516927 32589 leader_election.cc:304] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.517184 2080 raft_consensus.cc:2802] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.517288 2080 raft_consensus.cc:695] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:43.517580 2080 consensus_queue.cc:237] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.517951 952 rpcz_store.cc:276] 0623 08:04:37.851563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.851617 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:43.515242 (+5663625us) ts_tablet_manager.cc:1145] Deleting tablet 5629d11c44e241d598f3e519b2fe1a44
0623 08:04:43.515249 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.515299 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5649414}
I20250623 08:04:43.518311 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 297) took 4967 ms. Trace:
W20250623 08:04:43.518441 32453 consensus_peers.cc:487] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adc3cf4355c24b7395fcb22af8520346. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:43.516937 342 tablet_bootstrap.cc:654] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.519845 1953 consensus_queue.cc:1035] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.519881 1682 raft_consensus.cc:491] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.520043 1682 raft_consensus.cc:513] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.520825 1682 leader_election.cc:290] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.521598 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6fdea59f71e4a9d9fca8ede4b9439d2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.521868 32515 raft_consensus.cc:2391] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:43.521857 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6fdea59f71e4a9d9fca8ede4b9439d2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.522367 32314 leader_election.cc:304] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.522826 1843 raft_consensus.cc:3058] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.523757 343 tablet_bootstrap.cc:492] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.523874 343 ts_tablet_manager.cc:1397] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20250623 08:04:43.523969 339 tablet_bootstrap.cc:492] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.524086 339 ts_tablet_manager.cc:1397] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.130s user 0.007s sys 0.009s
I20250623 08:04:43.524605 343 raft_consensus.cc:357] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.524760 343 raft_consensus.cc:383] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.524801 343 raft_consensus.cc:738] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.524812 339 raft_consensus.cc:357] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.524946 339 raft_consensus.cc:383] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.524992 339 raft_consensus.cc:738] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.518399 952 rpcz_store.cc:276] 0623 08:04:38.550780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.570972 (+ 20192us) service_pool.cc:225] Handling call
0623 08:04:43.518174 (+4947202us) ts_tablet_manager.cc:1145] Deleting tablet 886fc0fa39c54bddb25ac01f68d95805
0623 08:04:43.518184 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.518263 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4885984}
I20250623 08:04:43.524991 343 consensus_queue.cc:260] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.525404 1843 raft_consensus.cc:2747] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.525514 952 tablet_replica.cc:331] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.525538 343 ts_tablet_manager.cc:1428] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:43.525811 343 tablet_bootstrap.cc:492] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.525800 952 raft_consensus.cc:2241] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:43.526947 32589 connection.cc:537] client connection to 127.31.69.62:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:43.527019 32708 connection.cc:537] client connection to 127.31.69.62:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250623 08:04:43.527297 981 tablet_replica.cc:331] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
W20250623 08:04:43.527493 32453 connection.cc:537] client connection to 127.31.69.62:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:43.528095 32708 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.31.69.62:39465)
W20250623 08:04:43.528550 32708 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.69.62:39465)
W20250623 08:04:43.528810 32708 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.31.69.62:39465)
W20250623 08:04:43.528983 32708 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.31.69.62:39465)
I20250623 08:04:43.529606 981 raft_consensus.cc:2241] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.529904 981 raft_consensus.cc:2270] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.530654 981 ts_tablet_manager.cc:1905] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:43.530869 32314 connection.cc:537] server connection from 127.0.0.1:35542 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250623 08:04:43.531006 32314 connection.cc:537] client connection to 127.31.69.62:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250623 08:04:43.531697 343 tablet_bootstrap.cc:654] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.535219 342 tablet_bootstrap.cc:492] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.535434 342 ts_tablet_manager.cc:1397] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20250623 08:04:43.536296 342 raft_consensus.cc:357] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.536574 342 raft_consensus.cc:383] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.536684 342 raft_consensus.cc:738] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.536971 342 consensus_queue.cc:260] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.537762 342 ts_tablet_manager.cc:1428] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:43.539039 342 tablet_bootstrap.cc:492] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.542132 979 ts_tablet_manager.cc:1905] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.545922 342 tablet_bootstrap.cc:654] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.556188 1957 raft_consensus.cc:491] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 08:04:43.556285 2128 negotiation.cc:337] Failed RPC negotiation. Trace:
0623 08:04:43.528596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.69.62:39465 (local address 127.0.0.1:42408)
0623 08:04:43.555894 (+ 27298us) negotiation.cc:107] Waiting for socket to connect
0623 08:04:43.555989 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.69.62:39465: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":27182,"threads_started":1}
I20250623 08:04:43.556533 1957 raft_consensus.cc:513] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.557271 1957 leader_election.cc:290] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:43.557428 385 leader_election.cc:336] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d31835ec4ad74332a00998d431492466 (127.31.69.62:39465): Network error: Client connection negotiation failed: client connection to 127.31.69.62:39465: connect: Connection reset by peer (error 104)
I20250623 08:04:43.557658 385 leader_election.cc:304] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678; no voters: 9752155c20ee42fbb7ff7b7d5b682444, d31835ec4ad74332a00998d431492466
I20250623 08:04:43.526681 339 consensus_queue.cc:260] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.559115 339 ts_tablet_manager.cc:1428] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20250623 08:04:43.559916 339 tablet_bootstrap.cc:492] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:43.564527 32453 connection.cc:537] server connection from 127.0.0.1:55526 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250623 08:04:43.566396 339 tablet_bootstrap.cc:654] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.566622 2136 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250623 08:04:43.567759 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11bb55088ab24b94af3e509810890247" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.569779 343 tablet_bootstrap.cc:492] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
W20250623 08:04:43.569928 32453 leader_election.cc:343] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.570173 343 ts_tablet_manager.cc:1397] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.008s
W20250623 08:04:43.571329 1213 connection.cc:537] client connection to 127.31.69.62:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250623 08:04:43.571632 343 raft_consensus.cc:357] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.571781 343 raft_consensus.cc:383] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.571830 343 raft_consensus.cc:738] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.573765 343 consensus_queue.cc:260] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.574230 343 ts_tablet_manager.cc:1428] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250623 08:04:43.574487 343 tablet_bootstrap.cc:492] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.575861 343 tablet_bootstrap.cc:654] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:43.582562 1213 leader_election.cc:336] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d31835ec4ad74332a00998d431492466 (127.31.69.62:39465): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250623 08:04:43.588413 952 raft_consensus.cc:2270] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.589236 342 tablet_bootstrap.cc:492] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.589386 342 ts_tablet_manager.cc:1397] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.006s
I20250623 08:04:43.589643 952 ts_tablet_manager.cc:1905] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.590030 457 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Refusing update from remote peer 9752155c20ee42fbb7ff7b7d5b682444: Log matching property violated. Preceding OpId in replica: term: 3 index: 760. Preceding OpId from leader: term: 4 index: 761. (index mismatch)
I20250623 08:04:43.590006 342 raft_consensus.cc:357] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.590144 342 raft_consensus.cc:383] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.590194 342 raft_consensus.cc:738] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.590389 342 consensus_queue.cc:260] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.590922 342 ts_tablet_manager.cc:1428] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250623 08:04:43.591166 342 tablet_bootstrap.cc:492] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.591154 2060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 761, Last known committed idx: 760, Time since last communication: 0.000s
I20250623 08:04:43.592501 342 tablet_bootstrap.cc:654] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.594635 979 ts_tablet_manager.cc:1918] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.594741 979 log.cc:1199] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/3740dd120745493688eeb4c1ac311dcb
I20250623 08:04:43.595130 979 ts_tablet_manager.cc:1939] T 3740dd120745493688eeb4c1ac311dcb P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:43.585023 892 master_runner.cc:186] Error getting master registration for 127.31.69.62:39465: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20250623 08:04:43.595598 892 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250623 08:04:43.595911 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 409) took 1297 ms. Trace:
I20250623 08:04:43.595993 979 rpcz_store.cc:276] 0623 08:04:42.298585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.303866 (+ 5281us) service_pool.cc:225] Handling call
0623 08:04:43.487502 (+1183636us) ts_tablet_manager.cc:1145] Deleting tablet 3740dd120745493688eeb4c1ac311dcb
0623 08:04:43.488497 (+ 995us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.549060 (+ 60563us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.594529 (+ 45469us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.595892 (+ 1363us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5248,"tablet-delete.queue_time_us":1183194}
I20250623 08:04:43.602566 343 tablet_bootstrap.cc:492] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.602842 343 ts_tablet_manager.cc:1397] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
W20250623 08:04:43.603924 32592 connection.cc:441] server connection from 127.0.0.1:36612 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 409) could send its response
I20250623 08:04:43.603992 979 tablet_replica.cc:331] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:43.604207 979 raft_consensus.cc:2241] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.606127 979 raft_consensus.cc:2270] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.606693 979 ts_tablet_manager.cc:1905] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.611510 342 tablet_bootstrap.cc:492] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.611686 342 ts_tablet_manager.cc:1397] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20250623 08:04:43.613605 979 ts_tablet_manager.cc:1918] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.613780 979 log.cc:1199] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/a8de2b1105564fdab386442b1de4cedb
I20250623 08:04:43.614193 979 ts_tablet_manager.cc:1939] T a8de2b1105564fdab386442b1de4cedb P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:43.615070 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 408) took 1316 ms. Trace:
I20250623 08:04:43.615198 979 rpcz_store.cc:276] 0623 08:04:42.298463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.306055 (+ 7592us) service_pool.cc:225] Handling call
0623 08:04:43.603940 (+1297885us) ts_tablet_manager.cc:1145] Deleting tablet a8de2b1105564fdab386442b1de4cedb
0623 08:04:43.603950 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.610296 (+ 6346us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.613502 (+ 3206us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.615044 (+ 1542us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1297643}
I20250623 08:04:43.615655 979 tablet_replica.cc:331] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:43.615862 979 raft_consensus.cc:2241] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.616052 979 raft_consensus.cc:2270] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.616603 979 ts_tablet_manager.cc:1905] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.616874 343 raft_consensus.cc:357] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.617082 343 raft_consensus.cc:383] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.617148 343 raft_consensus.cc:738] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:43.617439 32592 connection.cc:441] server connection from 127.0.0.1:36612 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 408) could send its response
I20250623 08:04:43.617357 343 consensus_queue.cc:260] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.618175 343 ts_tablet_manager.cc:1428] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250623 08:04:43.618818 343 tablet_bootstrap.cc:492] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.620226 343 tablet_bootstrap.cc:654] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.617780 342 raft_consensus.cc:357] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.626168 342 raft_consensus.cc:383] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.626281 342 raft_consensus.cc:738] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.626539 342 consensus_queue.cc:260] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.627739 342 ts_tablet_manager.cc:1428] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250623 08:04:43.628145 342 tablet_bootstrap.cc:492] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.628376 2136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9752155c20ee42fbb7ff7b7d5b682444. Latest consensus state: current_term: 4 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:43.628717 2136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:43.629648 342 tablet_bootstrap.cc:654] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.635599 952 ts_tablet_manager.cc:1918] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:43.635812 952 log.cc:1199] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f97f14cbaffb46d681d8b11c0213cae4
I20250623 08:04:43.636286 952 ts_tablet_manager.cc:1939] T f97f14cbaffb46d681d8b11c0213cae4 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.640528 1627 consensus_queue.cc:1035] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.643590 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_06e661ea03304faaaa16113c730a6d91 [id=60538f85a9b84a1190ef7403747476fb]
I20250623 08:04:43.644295 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1ced320c61ec4355bb2686a5c84a55eb [id=61c080721215406397c00aff27538c61]
I20250623 08:04:43.645206 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_afe210848a6b49b2a15d04cf79e1d47c [id=64522ca998ae445696a94c41ca58069a]
I20250623 08:04:43.646167 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_e2502c251f3c4a7f8caa4faab352ebda [id=664b85653cd341a6a6afe7d04fb9c2ac]
I20250623 08:04:43.646292 342 tablet_bootstrap.cc:492] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.646495 342 ts_tablet_manager.cc:1397] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20250623 08:04:43.647210 342 raft_consensus.cc:357] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.647377 342 raft_consensus.cc:383] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.647433 342 raft_consensus.cc:738] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.647523 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 310) took 5071 ms. Trace:
I20250623 08:04:43.648144 952 rpcz_store.cc:276] 0623 08:04:38.576154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.576223 (+ 69us) service_pool.cc:225] Handling call
0623 08:04:43.525425 (+4949202us) ts_tablet_manager.cc:1145] Deleting tablet f97f14cbaffb46d681d8b11c0213cae4
0623 08:04:43.525440 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.603552 (+ 78112us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.635468 (+ 31916us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.647500 (+ 12032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4862064}
W20250623 08:04:43.650111 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 310) could send its response
I20250623 08:04:43.647622 342 consensus_queue.cc:260] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.650748 342 ts_tablet_manager.cc:1428] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250623 08:04:43.651911 343 tablet_bootstrap.cc:492] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.652045 343 ts_tablet_manager.cc:1397] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.003s
I20250623 08:04:43.652675 343 raft_consensus.cc:357] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.652935 343 raft_consensus.cc:383] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.652995 343 raft_consensus.cc:738] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.653203 343 consensus_queue.cc:260] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.653801 343 ts_tablet_manager.cc:1428] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:43.654059 343 tablet_bootstrap.cc:492] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.655747 343 tablet_bootstrap.cc:654] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.573256 32372 raft_consensus.cc:3058] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.661525 343 tablet_bootstrap.cc:492] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.573573 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11bb55088ab24b94af3e509810890247" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.661301 1213 leader_election.cc:336] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d31835ec4ad74332a00998d431492466 (127.31.69.62:39465): Network error: Client connection negotiation failed: client connection to 127.31.69.62:39465: connect: Connection refused (error 111)
W20250623 08:04:43.662501 32453 leader_election.cc:343] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.662609 32453 leader_election.cc:304] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.586509 981 ts_tablet_manager.cc:1918] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250623 08:04:43.662890 981 log.cc:1199] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/8c01121f2e644d6aa6e8ec228bf75be9
I20250623 08:04:43.663283 981 ts_tablet_manager.cc:1939] T 8c01121f2e644d6aa6e8ec228bf75be9 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.663340 1953 raft_consensus.cc:2747] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.604439 32378 raft_consensus.cc:3058] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.665170 343 ts_tablet_manager.cc:1397] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20250623 08:04:43.668013 339 tablet_bootstrap.cc:492] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.668145 339 ts_tablet_manager.cc:1397] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.108s user 0.002s sys 0.001s
I20250623 08:04:43.669276 343 raft_consensus.cc:357] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.669508 343 raft_consensus.cc:383] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.669672 1957 raft_consensus.cc:491] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.669845 1957 raft_consensus.cc:513] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.670408 1957 leader_election.cc:290] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.671303 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92005abc61dd40ef9da8124cadc29b6a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.671483 32646 raft_consensus.cc:2466] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.671936 32453 leader_election.cc:304] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.672255 1957 raft_consensus.cc:2802] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.672339 1957 raft_consensus.cc:491] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.672415 1957 raft_consensus.cc:3058] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.675226 1957 raft_consensus.cc:513] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.676597 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92005abc61dd40ef9da8124cadc29b6a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.676755 32646 raft_consensus.cc:3058] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.680094 32646 raft_consensus.cc:2466] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.680258 1957 leader_election.cc:290] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.680716 32453 leader_election.cc:304] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.681010 1957 raft_consensus.cc:2802] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.681594 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92005abc61dd40ef9da8124cadc29b6a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.682102 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92005abc61dd40ef9da8124cadc29b6a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.682137 32453 leader_election.cc:343] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:43.682626 32453 leader_election.cc:343] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.682904 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 282) took 6663 ms. Trace:
I20250623 08:04:43.682986 981 rpcz_store.cc:276] 0623 08:04:37.019672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.030028 (+ 10356us) service_pool.cc:225] Handling call
0623 08:04:43.504165 (+6474137us) ts_tablet_manager.cc:1145] Deleting tablet 8c01121f2e644d6aa6e8ec228bf75be9
0623 08:04:43.515126 (+ 10961us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.561135 (+ 46009us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.586379 (+ 25244us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.682886 (+ 96507us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6473898}
I20250623 08:04:43.683372 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 284) took 6395 ms. Trace:
I20250623 08:04:43.683374 1957 raft_consensus.cc:695] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.683441 981 rpcz_store.cc:276] 0623 08:04:37.287609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.287669 (+ 60us) service_pool.cc:225] Handling call
0623 08:04:43.683266 (+6395597us) ts_tablet_manager.cc:1145] Deleting tablet 6b04e38d3da44301a450fe1fb5e2ada6
0623 08:04:43.683275 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.683355 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6395426}
I20250623 08:04:43.683686 981 tablet_replica.cc:331] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.683610 1957 consensus_queue.cc:237] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.683846 981 raft_consensus.cc:2241] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.684001 981 raft_consensus.cc:2270] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.684546 981 ts_tablet_manager.cc:1905] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:43.687659 1213 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 -> Peer d31835ec4ad74332a00998d431492466 (127.31.69.62:39465): Couldn't send request to peer d31835ec4ad74332a00998d431492466. Status: Network error: Client connection negotiation failed: client connection to 127.31.69.62:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:43.691169 343 raft_consensus.cc:738] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.691535 343 consensus_queue.cc:260] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.692307 343 ts_tablet_manager.cc:1428] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250623 08:04:43.692584 343 tablet_bootstrap.cc:492] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:43.690600 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 282) could send its response
W20250623 08:04:43.694118 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 284) could send its response
I20250623 08:04:43.695063 1910 raft_consensus.cc:491] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.695180 1910 raft_consensus.cc:513] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.695792 1910 leader_election.cc:290] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.695827 952 tablet_replica.cc:331] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.696014 952 raft_consensus.cc:2241] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.696353 952 raft_consensus.cc:2270] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.696583 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e17e1ca404b462da43aa1db7dc3b556" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.696797 32646 raft_consensus.cc:2391] T 8e17e1ca404b462da43aa1db7dc3b556 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:43.696939 952 ts_tablet_manager.cc:1905] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.698832 979 ts_tablet_manager.cc:1918] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:43.698942 979 log.cc:1199] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/a9ea74d20c5f4eefbcb8508ed1774026
I20250623 08:04:43.699275 979 ts_tablet_manager.cc:1939] T a9ea74d20c5f4eefbcb8508ed1774026 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:43.699168 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e17e1ca404b462da43aa1db7dc3b556" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.700742 1953 raft_consensus.cc:491] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.700874 1953 raft_consensus.cc:513] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.701118 32314 leader_election.cc:304] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.701395 1953 leader_election.cc:290] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.701670 1682 raft_consensus.cc:3058] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.701982 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b11777e0bd04e61a5bc8bb6ec4d97c3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.702147 32646 raft_consensus.cc:2466] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.702481 32453 leader_election.cc:304] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.702715 1953 raft_consensus.cc:2802] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.702805 1953 raft_consensus.cc:491] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.702862 1953 raft_consensus.cc:3058] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.704380 1682 raft_consensus.cc:2747] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.705024 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b11777e0bd04e61a5bc8bb6ec4d97c3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.705348 1953 raft_consensus.cc:513] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:43.705587 32453 leader_election.cc:343] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.705855 1953 leader_election.cc:290] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.706504 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]
I20250623 08:04:43.706507 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b11777e0bd04e61a5bc8bb6ec4d97c3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.707172 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]
W20250623 08:04:43.708223 32453 leader_election.cc:343] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.708523 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b11777e0bd04e61a5bc8bb6ec4d97c3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.708678 32646 raft_consensus.cc:3058] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.708729 979 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 410) took 1365 ms. Trace:
I20250623 08:04:43.710810 979 rpcz_store.cc:276] 0623 08:04:42.343434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.343569 (+ 135us) service_pool.cc:225] Handling call
0623 08:04:43.615559 (+1271990us) ts_tablet_manager.cc:1145] Deleting tablet a9ea74d20c5f4eefbcb8508ed1774026
0623 08:04:43.615571 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.644815 (+ 29244us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.698661 (+ 53846us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.708714 (+ 10053us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1271736}
W20250623 08:04:43.711057 32592 connection.cc:441] server connection from 127.0.0.1:36612 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36612 (request call id 410) could send its response
I20250623 08:04:43.711374 32646 raft_consensus.cc:2466] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.711742 32453 leader_election.cc:304] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.711930 343 tablet_bootstrap.cc:654] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.712059 1627 raft_consensus.cc:2802] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.712472 1627 raft_consensus.cc:695] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.712781 1627 consensus_queue.cc:237] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.717227 2152 tablet.cc:1722] T f8a5e35c1974456b8079ab8907c2652e P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:43.718080 1910 raft_consensus.cc:491] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.719596 1910 raft_consensus.cc:513] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.720178 1910 leader_election.cc:290] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.722268 339 raft_consensus.cc:357] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.722440 339 raft_consensus.cc:383] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.722379 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c14e325fa5f4f848748c346bc25dde0" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.722514 339 raft_consensus.cc:738] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.722720 339 consensus_queue.cc:260] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:43.722913 32453 consensus_peers.cc:487] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:43.722949 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c14e325fa5f4f848748c346bc25dde0" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.723258 32646 raft_consensus.cc:2391] T 3c14e325fa5f4f848748c346bc25dde0 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:43.723469 339 ts_tablet_manager.cc:1428] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.055s user 0.002s sys 0.000s
I20250623 08:04:43.723733 339 tablet_bootstrap.cc:492] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.724462 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_63b537810a3a4f478e4226cabee7004d [id=692d14d5e7224e37b0a78ac3171bfefa]
I20250623 08:04:43.725116 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297]
I20250623 08:04:43.725654 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]
I20250623 08:04:43.727615 32314 leader_election.cc:304] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.729351 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]
I20250623 08:04:43.729874 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca]
I20250623 08:04:43.730373 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_91d48bbc63414bb4a1fb90f438f52eba [id=6eafeaf2ab324e3ebcf4ad4ad168cb82]
I20250623 08:04:43.730440 1910 raft_consensus.cc:3058] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.731643 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0]
I20250623 08:04:43.732126 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_992eb98f4db0454083b215b3ac3d38b7 [id=743dc03385c4498c8c1ca8df7e859bee]
I20250623 08:04:43.732573 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_87e0e2fcec324a53ae2cfd5b3156e071 [id=75e5879d4117409da29206bb682898cc]
I20250623 08:04:43.733269 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]
I20250623 08:04:43.733408 1910 raft_consensus.cc:2747] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.733615 1823 raft_consensus.cc:491] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.733762 1823 raft_consensus.cc:513] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.734315 1823 leader_election.cc:290] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.735508 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea77b97018b74708bd6057bd2d65f90b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.735693 32654 raft_consensus.cc:2466] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.736104 32453 leader_election.cc:304] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.736335 1953 raft_consensus.cc:2802] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.736410 1953 raft_consensus.cc:491] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.736462 1953 raft_consensus.cc:3058] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.739125 1953 raft_consensus.cc:513] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.739619 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea77b97018b74708bd6057bd2d65f90b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.739710 1953 leader_election.cc:290] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.740104 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea77b97018b74708bd6057bd2d65f90b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.740298 32654 raft_consensus.cc:3058] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.742089 343 tablet_bootstrap.cc:492] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.742239 343 ts_tablet_manager.cc:1397] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.001s
I20250623 08:04:43.742990 981 ts_tablet_manager.cc:1918] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:43.743070 32654 raft_consensus.cc:2466] T ea77b97018b74708bd6057bd2d65f90b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.743140 343 raft_consensus.cc:357] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.743332 343 raft_consensus.cc:383] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.743384 343 raft_consensus.cc:738] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.743535 32453 leader_election.cc:304] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.743775 1953 raft_consensus.cc:2802] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.744189 1953 raft_consensus.cc:695] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.744427 1953 consensus_queue.cc:237] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.745366 981 log.cc:1199] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/c0773fc60d4a463596f536f15d58bf57
I20250623 08:04:43.745837 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea77b97018b74708bd6057bd2d65f90b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:43.746388 32453 leader_election.cc:343] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.746688 981 ts_tablet_manager.cc:1939] T c0773fc60d4a463596f536f15d58bf57 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:43.746791 32453 leader_election.cc:343] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.748070 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 295) took 6110 ms. Trace:
I20250623 08:04:43.752254 952 ts_tablet_manager.cc:1918] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:43.752651 952 log.cc:1199] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/025365d2fa0a43dd983d395a05cc2af1
I20250623 08:04:43.753324 952 ts_tablet_manager.cc:1939] T 025365d2fa0a43dd983d395a05cc2af1 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.756222 2136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:43.756454 2136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [sys.catalog]: This master's current role is: FOLLOWER
I20250623 08:04:43.757902 342 tablet_bootstrap.cc:492] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.758018 1957 raft_consensus.cc:491] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.758147 1957 raft_consensus.cc:513] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.758697 1957 leader_election.cc:290] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.759042 343 consensus_queue.cc:260] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.759655 342 tablet_bootstrap.cc:654] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.762823 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8111e6117c1541ab8905d5a5a7e37292" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.763450 32646 raft_consensus.cc:2466] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.764060 32453 leader_election.cc:304] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:43.766924 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 311) took 5190 ms. Trace:
I20250623 08:04:43.770807 981 rpcz_store.cc:276] 0623 08:04:37.637502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.637865 (+ 363us) service_pool.cc:225] Handling call
0623 08:04:43.683650 (+6045785us) ts_tablet_manager.cc:1145] Deleting tablet c0773fc60d4a463596f536f15d58bf57
0623 08:04:43.683658 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.706940 (+ 23282us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.742885 (+ 35945us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.748046 (+ 5161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6045570}
W20250623 08:04:43.771091 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 295) could send its response
I20250623 08:04:43.744903 339 tablet_bootstrap.cc:654] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.762843 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8111e6117c1541ab8905d5a5a7e37292" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.772362 343 ts_tablet_manager.cc:1428] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20250623 08:04:43.772953 981 tablet_replica.cc:331] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.773212 981 raft_consensus.cc:2241] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.773413 981 raft_consensus.cc:2270] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.774516 981 ts_tablet_manager.cc:1905] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.775314 952 rpcz_store.cc:276] 0623 08:04:38.576519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.576570 (+ 51us) service_pool.cc:225] Handling call
0623 08:04:43.651022 (+5074452us) ts_tablet_manager.cc:1145] Deleting tablet 025365d2fa0a43dd983d395a05cc2af1
0623 08:04:43.695761 (+ 44739us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.701732 (+ 5971us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.710642 (+ 8910us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.766902 (+ 56260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4987592}
I20250623 08:04:43.775885 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 326) took 4461 ms. Trace:
I20250623 08:04:43.776068 952 rpcz_store.cc:276] 0623 08:04:39.314277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.314333 (+ 56us) service_pool.cc:225] Handling call
0623 08:04:43.775752 (+4461419us) ts_tablet_manager.cc:1145] Deleting tablet c74e8c697d244ac2ac8174bb1fc76e61
0623 08:04:43.775764 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.775868 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4461211}
I20250623 08:04:43.779138 952 tablet_replica.cc:331] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.779584 952 raft_consensus.cc:2241] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.779983 952 raft_consensus.cc:2270] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.780656 952 ts_tablet_manager.cc:1905] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.771953 1957 raft_consensus.cc:2802] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250623 08:04:43.772274 32453 leader_election.cc:343] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.774086 1953 raft_consensus.cc:491] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.783025 1953 raft_consensus.cc:513] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:43.783245 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 311) could send its response
W20250623 08:04:43.783468 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 326) could send its response
I20250623 08:04:43.783825 1823 raft_consensus.cc:491] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.783941 1823 raft_consensus.cc:513] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.784381 1823 leader_election.cc:290] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.784732 1953 leader_election.cc:290] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.783799 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_7c712a75a62f41dba90fcf504f4f752c [id=780a3bc4b3004e06a840a0b63e0c37c6]
I20250623 08:04:43.785111 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cac83131e6854890b803b3d89ded7cf8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.785373 32654 raft_consensus.cc:2466] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:43.785617 343 tablet_bootstrap.cc:492] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.786120 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0107b41d5c04a5c818f6792f2c18ce5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.786211 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cac83131e6854890b803b3d89ded7cf8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.787180 32453 leader_election.cc:343] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:43.787396 32453 leader_election.cc:343] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.787577 32453 leader_election.cc:304] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.787801 1953 raft_consensus.cc:2802] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.787876 1953 raft_consensus.cc:491] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.787933 1953 raft_consensus.cc:3058] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.788157 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0107b41d5c04a5c818f6792f2c18ce5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:43.788700 32453 leader_election.cc:343] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.788797 32453 leader_election.cc:304] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.789062 343 tablet_bootstrap.cc:654] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.789054 1957 raft_consensus.cc:491] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.789146 1957 raft_consensus.cc:3058] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.789925 1359 consensus_queue.cc:1035] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.790521 1359 raft_consensus.cc:2747] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.792097 1953 raft_consensus.cc:513] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.792912 1953 leader_election.cc:290] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.793469 2060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "9752155c20ee42fbb7ff7b7d5b682444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } } }
I20250623 08:04:43.796213 2060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 [sys.catalog]: This master's current role is: LEADER
I20250623 08:04:43.796394 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]
I20250623 08:04:43.797478 339 tablet_bootstrap.cc:492] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.797602 339 ts_tablet_manager.cc:1397] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.001s
I20250623 08:04:43.798338 339 raft_consensus.cc:357] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.798478 339 raft_consensus.cc:383] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.798523 339 raft_consensus.cc:738] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.798728 339 consensus_queue.cc:260] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.802106 1910 raft_consensus.cc:491] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.802266 1910 raft_consensus.cc:513] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.802886 1910 leader_election.cc:290] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.803215 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd7f3d2e67e84bfcbd45a21658f69e7b" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.804302 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1a60513fe9e94660aa640b5522394bbb [id=79c8424b812d4c98b8dbd8e1fdd5dd52]
I20250623 08:04:43.807103 339 ts_tablet_manager.cc:1428] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250623 08:04:43.807397 339 tablet_bootstrap.cc:492] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.809094 339 tablet_bootstrap.cc:654] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.808974 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_ea4d110c161f4b008acde5f22ffa3562 [id=79d117a74f18466ab87a1701e641e7da]
I20250623 08:04:43.809911 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd7f3d2e67e84bfcbd45a21658f69e7b" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.813601 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_7a30e48550d443ccbe36157dd0e35bcc [id=7a1f259c8fc24b4ca83cfa09e5673d58]
I20250623 08:04:43.817914 32314 leader_election.cc:304] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.818202 1910 raft_consensus.cc:2747] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.819106 981 ts_tablet_manager.cc:1918] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:43.819264 981 log.cc:1199] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/f9f4829e103b460eb582beff3b966bb4
I20250623 08:04:43.819622 981 ts_tablet_manager.cc:1939] T f9f4829e103b460eb582beff3b966bb4 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.820470 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 296) took 6156 ms. Trace:
I20250623 08:04:43.820276 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]
I20250623 08:04:43.820597 981 rpcz_store.cc:276] 0623 08:04:37.664368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.665889 (+ 1521us) service_pool.cc:225] Handling call
0623 08:04:43.772904 (+6107015us) ts_tablet_manager.cc:1145] Deleting tablet f9f4829e103b460eb582beff3b966bb4
0623 08:04:43.772914 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.815663 (+ 42749us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.819013 (+ 3350us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.820453 (+ 1440us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6106846}
W20250623 08:04:43.820902 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 296) could send its response
I20250623 08:04:43.821153 981 tablet_replica.cc:331] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.821350 981 raft_consensus.cc:2241] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.821506 981 raft_consensus.cc:2270] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.822002 981 ts_tablet_manager.cc:1905] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.823565 339 tablet_bootstrap.cc:492] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.823701 339 ts_tablet_manager.cc:1397] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20250623 08:04:43.823933 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e [id=7a569a9fe3bd4c608ddca48a734df1da]
I20250623 08:04:43.824360 339 raft_consensus.cc:357] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.824508 339 raft_consensus.cc:383] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.824563 339 raft_consensus.cc:738] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.824738 339 consensus_queue.cc:260] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.827811 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_9a49c84f75b84594ab81fcf31da3991f [id=7b34109f27f5441e855b23fb253a8e99]
I20250623 08:04:43.828609 339 ts_tablet_manager.cc:1428] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:43.830065 343 tablet_bootstrap.cc:492] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.830193 343 ts_tablet_manager.cc:1397] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20250623 08:04:43.830921 343 raft_consensus.cc:357] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.831086 343 raft_consensus.cc:383] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.831147 343 raft_consensus.cc:738] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.831336 343 consensus_queue.cc:260] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.831746 343 ts_tablet_manager.cc:1428] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250623 08:04:43.831983 343 tablet_bootstrap.cc:492] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.833513 343 tablet_bootstrap.cc:654] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.833850 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_161ad5ddd1cb40e0989e5c237a205db1 [id=7cc0919082d04fc4a02aa856a42e5529]
I20250623 08:04:43.834570 981 ts_tablet_manager.cc:1918] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:43.835456 1957 raft_consensus.cc:513] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.835129 339 tablet_bootstrap.cc:492] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.836898 1957 leader_election.cc:290] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.837358 339 tablet_bootstrap.cc:654] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.838075 981 log.cc:1199] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/2c5ecd4fa5574a1b90b6e82f89ee88f7
I20250623 08:04:43.838415 981 ts_tablet_manager.cc:1939] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:43.841377 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 52679.7us
I20250623 08:04:43.842862 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8111e6117c1541ab8905d5a5a7e37292" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.843317 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cac83131e6854890b803b3d89ded7cf8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.844136 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8111e6117c1541ab8905d5a5a7e37292" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.844301 32647 raft_consensus.cc:3058] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:43.844326 32453 leader_election.cc:343] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:43.844800 32453 leader_election.cc:343] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.845265 1957 raft_consensus.cc:491] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.845443 1957 raft_consensus.cc:513] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.845681 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 297) took 6136 ms. Trace:
I20250623 08:04:43.845757 981 rpcz_store.cc:276] 0623 08:04:37.709677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.709729 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:43.821112 (+6111383us) ts_tablet_manager.cc:1145] Deleting tablet 2c5ecd4fa5574a1b90b6e82f89ee88f7
0623 08:04:43.821121 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.825169 (+ 4048us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.834474 (+ 9305us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.845658 (+ 11184us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6111184}
W20250623 08:04:43.845955 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 297) could send its response
I20250623 08:04:43.846292 1957 leader_election.cc:290] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.847373 981 tablet_replica.cc:331] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.847554 981 raft_consensus.cc:2241] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.847841 981 raft_consensus.cc:2270] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.848248 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cac83131e6854890b803b3d89ded7cf8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:43.848424 32654 raft_consensus.cc:3058] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.849817 981 ts_tablet_manager.cc:1905] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.850927 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34da529ffde4595a23fc6596b30e3e9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.851465 32453 leader_election.cc:343] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.851189 952 ts_tablet_manager.cc:1918] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.851706 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34da529ffde4595a23fc6596b30e3e9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.851794 952 log.cc:1199] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/724d662bcf754c74a8f3dff91ee258ae
W20250623 08:04:43.853271 32453 leader_election.cc:343] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.853363 32453 leader_election.cc:304] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.853699 1957 raft_consensus.cc:2747] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.853967 952 ts_tablet_manager.cc:1939] T 724d662bcf754c74a8f3dff91ee258ae P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.854786 2013 raft_consensus.cc:491] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.854961 32647 raft_consensus.cc:2466] T 8111e6117c1541ab8905d5a5a7e37292 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.854938 2013 raft_consensus.cc:513] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.855468 2013 leader_election.cc:290] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.855684 1910 raft_consensus.cc:491] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.855810 1910 raft_consensus.cc:513] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.856046 1957 consensus_queue.cc:1035] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.856142 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b04e38d3da44301a450fe1fb5e2ada6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.856271 32518 raft_consensus.cc:2466] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:43.856326 1910 leader_election.cc:290] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.856895 32654 raft_consensus.cc:2466] T cac83131e6854890b803b3d89ded7cf8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:43.857092 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16bc969d09f74b0986070a504f9f670e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.857565 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b04e38d3da44301a450fe1fb5e2ada6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.858166 32589 leader_election.cc:343] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.858352 32589 leader_election.cc:304] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.858976 2013 raft_consensus.cc:2802] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.859073 2013 raft_consensus.cc:491] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.859148 2013 raft_consensus.cc:3058] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.859187 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 325) took 4544 ms. Trace:
I20250623 08:04:43.859593 952 rpcz_store.cc:276] 0623 08:04:39.314169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.314944 (+ 775us) service_pool.cc:225] Handling call
0623 08:04:43.779088 (+4464144us) ts_tablet_manager.cc:1145] Deleting tablet 724d662bcf754c74a8f3dff91ee258ae
0623 08:04:43.779100 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.805884 (+ 26784us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.850624 (+ 44740us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.859000 (+ 8376us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4463894}
I20250623 08:04:43.861781 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 327) took 4546 ms. Trace:
I20250623 08:04:43.862175 952 rpcz_store.cc:276] 0623 08:04:39.314788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.315227 (+ 439us) service_pool.cc:225] Handling call
0623 08:04:43.861186 (+4545959us) ts_tablet_manager.cc:1145] Deleting tablet 59015204545e436cb558fe4465a67c97
0623 08:04:43.861197 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.861607 (+ 410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4545621}
I20250623 08:04:43.863863 32453 leader_election.cc:304] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.864120 2160 raft_consensus.cc:2802] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.864240 2160 raft_consensus.cc:695] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.864290 342 tablet_bootstrap.cc:492] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.864261 32453 leader_election.cc:304] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.864395 342 ts_tablet_manager.cc:1397] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.107s user 0.010s sys 0.007s
I20250623 08:04:43.864532 1627 raft_consensus.cc:2802] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.864485 2160 consensus_queue.cc:237] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.865041 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16bc969d09f74b0986070a504f9f670e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.865547 32314 leader_election.cc:304] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:43.862666 2013 raft_consensus.cc:513] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.865641 952 tablet_replica.cc:331] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.865801 1910 raft_consensus.cc:2747] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.866950 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b04e38d3da44301a450fe1fb5e2ada6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:43.866883 952 raft_consensus.cc:2241] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
W20250623 08:04:43.867523 32589 leader_election.cc:343] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.867926 2013 leader_election.cc:290] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.868753 952 raft_consensus.cc:2270] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.871256 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_cace1e64be664c9c81b588e80d704aaa [id=7ed25123fbc749909d1d90134be64278]
I20250623 08:04:43.872153 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_315a89d642a74362b8a04f8b82edd22f [id=7f7d25e8c0904f09b3cbd93034adcb0d]
I20250623 08:04:43.872664 952 ts_tablet_manager.cc:1905] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.878935 343 tablet_bootstrap.cc:492] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.879073 343 ts_tablet_manager.cc:1397] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.002s
I20250623 08:04:43.879848 343 raft_consensus.cc:357] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.880010 343 raft_consensus.cc:383] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.880064 343 raft_consensus.cc:738] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:43.864650 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 325) could send its response
I20250623 08:04:43.864650 1627 raft_consensus.cc:695] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:43.865053 342 raft_consensus.cc:357] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:43.881130 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 327) could send its response
I20250623 08:04:43.881132 342 raft_consensus.cc:383] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.881243 342 raft_consensus.cc:738] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.881270 1627 consensus_queue.cc:237] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.881413 342 consensus_queue.cc:260] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:43.882031 342 ts_tablet_manager.cc:1428] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250623 08:04:43.882313 342 tablet_bootstrap.cc:492] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.882421 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_06b731510ae743608f1fbc4d250acf14 [id=7fbe15151173464ea0ee480720c58b57]
I20250623 08:04:43.882377 343 consensus_queue.cc:260] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.883090 343 ts_tablet_manager.cc:1428] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250623 08:04:43.883406 343 tablet_bootstrap.cc:492] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.884037 342 tablet_bootstrap.cc:654] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.885331 343 tablet_bootstrap.cc:654] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.892500 981 ts_tablet_manager.cc:1918] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.895104 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b04e38d3da44301a450fe1fb5e2ada6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:43.895274 32518 raft_consensus.cc:3058] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.897622 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_c491a58f8bb24bbf85189ea654bdd45f [id=826cd9624f2a4f818f02427a9462a9b2]
I20250623 08:04:43.897898 32518 raft_consensus.cc:2466] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:43.898141 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91]
I20250623 08:04:43.898394 981 log.cc:1199] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/f83f68ebda7e4826bbb17278cd4ac5a8
I20250623 08:04:43.898854 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_b91721eeaabb4377b4d62483ad065bee [id=8544c855ea7e4fed887765c7730c3ccd]
I20250623 08:04:43.899295 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]
I20250623 08:04:43.899770 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]
I20250623 08:04:43.900125 981 ts_tablet_manager.cc:1939] T f83f68ebda7e4826bbb17278cd4ac5a8 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.900185 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_0bb2584ec3e0467194a9591bbe5ae477 [id=87e7893d85264d4c9cbcc5564f840390]
I20250623 08:04:43.900621 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]
I20250623 08:04:43.901026 2013 raft_consensus.cc:491] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.901069 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_eda413af2cf84c009c8fc4b375b57e2a [id=898ec01859bf4ba29e38b4eaf3d5392e]
I20250623 08:04:43.901161 2013 raft_consensus.cc:513] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.901469 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_c67b6294376343b38de7a2aa4ed7bcea [id=898fc58a25554c09867e70728f4a7508]
I20250623 08:04:43.901908 2013 leader_election.cc:290] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.902130 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_fe81102858f244af9a4a286eaf6e34bf [id=8a0d9f5619d843148169561ffc7cfdc0]
I20250623 08:04:43.902647 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_f933517ba72b4e44bd0432107e9a9f9f [id=8a7b2a641e2f4e638cef8b6604ad12f1]
I20250623 08:04:43.904033 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89bb7ea31b4d4ce7a3c063853142fbb2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:43.904579 32589 leader_election.cc:343] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.904850 343 tablet_bootstrap.cc:492] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.905490 343 ts_tablet_manager.cc:1397] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20250623 08:04:43.905735 342 tablet_bootstrap.cc:492] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.905994 342 ts_tablet_manager.cc:1397] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20250623 08:04:43.906301 32589 leader_election.cc:304] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.906623 2013 raft_consensus.cc:2802] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:43.906733 2013 raft_consensus.cc:695] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:43.906968 2013 consensus_queue.cc:237] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.907163 342 raft_consensus.cc:357] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:43.907423 342 raft_consensus.cc:383] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.907547 342 raft_consensus.cc:738] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.907971 2080 consensus_queue.cc:1035] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.907871 342 consensus_queue.cc:260] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:43.908797 342 ts_tablet_manager.cc:1428] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:43.909286 342 tablet_bootstrap.cc:492] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.910508 343 raft_consensus.cc:357] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.911230 2160 raft_consensus.cc:491] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.911373 2160 raft_consensus.cc:513] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:43.911885 2160 leader_election.cc:290] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.912065 343 raft_consensus.cc:383] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.912261 343 raft_consensus.cc:738] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.912748 343 consensus_queue.cc:260] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.913548 343 ts_tablet_manager.cc:1428] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250623 08:04:43.914419 343 tablet_bootstrap.cc:492] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.916715 343 tablet_bootstrap.cc:654] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.927874 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 298) took 6215 ms. Trace:
I20250623 08:04:43.928113 981 rpcz_store.cc:276] 0623 08:04:37.711915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:37.711964 (+ 49us) service_pool.cc:225] Handling call
0623 08:04:43.847018 (+6135054us) ts_tablet_manager.cc:1145] Deleting tablet f83f68ebda7e4826bbb17278cd4ac5a8
0623 08:04:43.847026 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.880531 (+ 33505us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.892394 (+ 11863us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.927847 (+ 35453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6134876}
W20250623 08:04:43.928504 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 298) could send its response
I20250623 08:04:43.928916 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 302) took 5923 ms. Trace:
I20250623 08:04:43.929705 981 rpcz_store.cc:276] 0623 08:04:38.005075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.005133 (+ 58us) service_pool.cc:225] Handling call
0623 08:04:43.928787 (+5923654us) ts_tablet_manager.cc:1145] Deleting tablet 886fc0fa39c54bddb25ac01f68d95805
0623 08:04:43.928797 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.928898 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5885686}
W20250623 08:04:43.929987 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 302) could send its response
I20250623 08:04:43.930298 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 315) took 5470 ms. Trace:
I20250623 08:04:43.930397 981 rpcz_store.cc:276] 0623 08:04:38.459631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.459839 (+ 208us) service_pool.cc:225] Handling call
0623 08:04:43.930196 (+5470357us) ts_tablet_manager.cc:1145] Deleting tablet f97f14cbaffb46d681d8b11c0213cae4
0623 08:04:43.930204 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.930286 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5470154}
W20250623 08:04:43.930636 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 315) could send its response
I20250623 08:04:43.930922 981 tablet_replica.cc:331] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.931131 981 raft_consensus.cc:2241] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.931336 981 raft_consensus.cc:2270] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.931916 981 ts_tablet_manager.cc:1905] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.912407 342 tablet_bootstrap.cc:654] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.912595 1957 consensus_queue.cc:1035] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20250623 08:04:43.937328 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89bb7ea31b4d4ce7a3c063853142fbb2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:43.937520 32515 raft_consensus.cc:2466] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:43.938406 981 ts_tablet_manager.cc:1918] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:43.938505 981 log.cc:1199] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/025365d2fa0a43dd983d395a05cc2af1
I20250623 08:04:43.938926 981 ts_tablet_manager.cc:1939] T 025365d2fa0a43dd983d395a05cc2af1 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.940109 339 tablet_bootstrap.cc:492] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:43.940228 339 ts_tablet_manager.cc:1397] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.012s
I20250623 08:04:43.940933 339 raft_consensus.cc:357] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.941085 339 raft_consensus.cc:383] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.941133 339 raft_consensus.cc:738] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.941347 339 consensus_queue.cc:260] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.945473 339 ts_tablet_manager.cc:1428] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:43.946300 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 316) took 5486 ms. Trace:
I20250623 08:04:43.948195 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b61b02cf2904404f8c0ffef7d268fefc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.950402 952 ts_tablet_manager.cc:1918] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.950634 952 log.cc:1199] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/bfd3e263464c4f2c8af9262acf211193
I20250623 08:04:43.951393 952 ts_tablet_manager.cc:1939] T bfd3e263464c4f2c8af9262acf211193 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:43.947216 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b61b02cf2904404f8c0ffef7d268fefc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.951429 981 rpcz_store.cc:276] 0623 08:04:38.459839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.460076 (+ 237us) service_pool.cc:225] Handling call
0623 08:04:43.930882 (+5470806us) ts_tablet_manager.cc:1145] Deleting tablet 025365d2fa0a43dd983d395a05cc2af1
0623 08:04:43.930891 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.935511 (+ 4620us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.938319 (+ 2808us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.946282 (+ 7963us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5470662}
W20250623 08:04:43.951798 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 316) could send its response
I20250623 08:04:43.951365 339 tablet_bootstrap.cc:492] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:43.952956 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 332) took 4629 ms. Trace:
I20250623 08:04:43.953289 339 tablet_bootstrap.cc:654] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.955240 343 tablet_bootstrap.cc:492] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.955422 343 ts_tablet_manager.cc:1397] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20250623 08:04:43.956215 343 raft_consensus.cc:357] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.956426 343 raft_consensus.cc:383] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.956527 343 raft_consensus.cc:738] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.956820 343 consensus_queue.cc:260] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.957263 32589 leader_election.cc:304] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.957363 343 ts_tablet_manager.cc:1428] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:43.957634 343 tablet_bootstrap.cc:492] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:43.958109 981 tablet_replica.cc:331] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.958348 981 raft_consensus.cc:2241] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.958556 981 raft_consensus.cc:2270] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.960255 343 tablet_bootstrap.cc:654] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.960911 981 ts_tablet_manager.cc:1905] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.960266 2013 raft_consensus.cc:2802] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:43.962553 2013 raft_consensus.cc:491] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:43.962661 2013 raft_consensus.cc:3058] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:43.963326 32453 leader_election.cc:343] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.965390 2013 raft_consensus.cc:513] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.965811 342 tablet_bootstrap.cc:492] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:43.965941 342 ts_tablet_manager.cc:1397] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20250623 08:04:43.965966 2013 leader_election.cc:290] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:43.966614 342 raft_consensus.cc:357] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:43.966811 342 raft_consensus.cc:383] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.966871 342 raft_consensus.cc:738] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.967089 342 consensus_queue.cc:260] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:43.967245 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89bb7ea31b4d4ce7a3c063853142fbb2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:43.967897 32589 leader_election.cc:343] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.968379 342 ts_tablet_manager.cc:1428] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250623 08:04:43.968940 342 tablet_bootstrap.cc:492] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:43.971674 1627 consensus_queue.cc:1035] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.972843 342 tablet_bootstrap.cc:654] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.973278 343 tablet_bootstrap.cc:492] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:43.973822 343 ts_tablet_manager.cc:1397] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250623 08:04:43.953171 952 rpcz_store.cc:276] 0623 08:04:39.323582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.323616 (+ 34us) service_pool.cc:225] Handling call
0623 08:04:43.865417 (+4541801us) ts_tablet_manager.cc:1145] Deleting tablet bfd3e263464c4f2c8af9262acf211193
0623 08:04:43.865431 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.918319 (+ 52888us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.950216 (+ 31897us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.952932 (+ 2716us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4541434}
I20250623 08:04:43.975539 343 raft_consensus.cc:357] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.975970 343 raft_consensus.cc:383] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.975838 952 tablet_replica.cc:331] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:43.976171 343 raft_consensus.cc:738] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:43.976364 343 consensus_queue.cc:260] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.976853 343 ts_tablet_manager.cc:1428] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250623 08:04:43.976531 32453 leader_election.cc:343] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.977030 32453 leader_election.cc:304] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.977142 343 tablet_bootstrap.cc:492] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:43.977360 32589 consensus_peers.cc:487] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:43.978549 1957 raft_consensus.cc:2747] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.978864 343 tablet_bootstrap.cc:654] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:43.979079 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_cddb0cb673ab4aacb6e69f181369ca8a [id=8afa0fb6b40242fb826b52206a1173c1]
I20250623 08:04:43.978339 952 raft_consensus.cc:2241] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:43.983345 981 ts_tablet_manager.cc:1918] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:43.983705 981 log.cc:1199] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/724d662bcf754c74a8f3dff91ee258ae
I20250623 08:04:43.983870 1957 raft_consensus.cc:491] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:43.984002 1957 raft_consensus.cc:513] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.984483 981 ts_tablet_manager.cc:1939] T 724d662bcf754c74a8f3dff91ee258ae P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:43.984541 1957 leader_election.cc:290] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:43.984905 952 pending_rounds.cc:70] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:43.985008 952 pending_rounds.cc:77] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727457105321984 op_type: NO_OP noop_request { }
I20250623 08:04:43.985153 952 raft_consensus.cc:2887] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:43.985294 952 raft_consensus.cc:2270] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:43.986215 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 332) could send its response
I20250623 08:04:43.986791 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 330) took 5136 ms. Trace:
I20250623 08:04:43.986913 981 rpcz_store.cc:276] 0623 08:04:38.850778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.852206 (+ 1428us) service_pool.cc:225] Handling call
0623 08:04:43.957825 (+5105619us) ts_tablet_manager.cc:1145] Deleting tablet 724d662bcf754c74a8f3dff91ee258ae
0623 08:04:43.957834 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.965334 (+ 7500us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.981452 (+ 16118us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:43.986763 (+ 5311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5105423}
I20250623 08:04:43.987555 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_6b3f6aad5faf4f2a88573140691c6291 [id=8e322f09172b40a8ba5aea16337fb020]
I20250623 08:04:43.988417 343 tablet_bootstrap.cc:492] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
W20250623 08:04:43.987453 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 330) could send its response
I20250623 08:04:43.989130 32380 raft_consensus.cc:3058] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.989166 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dbbdbebdf3b4b50bda31a94fc0de36f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:43.987332 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 331) took 5136 ms. Trace:
I20250623 08:04:43.990067 981 rpcz_store.cc:276] 0623 08:04:38.850883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.852440 (+ 1557us) service_pool.cc:225] Handling call
0623 08:04:43.987233 (+5134793us) ts_tablet_manager.cc:1145] Deleting tablet c74e8c697d244ac2ac8174bb1fc76e61
0623 08:04:43.987241 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.987320 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5134651}
I20250623 08:04:43.990438 952 ts_tablet_manager.cc:1905] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:43.990489 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 331) could send its response
I20250623 08:04:43.990861 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 333) took 5139 ms. Trace:
I20250623 08:04:43.988701 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dbbdbebdf3b4b50bda31a94fc0de36f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:43.991168 343 ts_tablet_manager.cc:1397] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250623 08:04:43.991485 2160 consensus_queue.cc:1035] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.991698 981 rpcz_store.cc:276] 0623 08:04:38.851137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.853283 (+ 2146us) service_pool.cc:225] Handling call
0623 08:04:43.990740 (+5137457us) ts_tablet_manager.cc:1145] Deleting tablet 59015204545e436cb558fe4465a67c97
0623 08:04:43.990772 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.990847 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5137289}
W20250623 08:04:43.992147 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 333) could send its response
I20250623 08:04:43.992331 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89bb7ea31b4d4ce7a3c063853142fbb2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:43.992511 32515 raft_consensus.cc:3058] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:43.992794 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 334) took 5059 ms. Trace:
I20250623 08:04:43.993341 343 raft_consensus.cc:357] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.993585 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_070eb9d8f3af4ed9bb823cf579ade9c8 [id=8f90ed2f56464b9f8dc494452e6c6aed]
I20250623 08:04:43.993682 981 rpcz_store.cc:276] 0623 08:04:38.933661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.934070 (+ 409us) service_pool.cc:225] Handling call
0623 08:04:43.992689 (+5058619us) ts_tablet_manager.cc:1145] Deleting tablet 92005abc61dd40ef9da8124cadc29b6a
0623 08:04:43.992697 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.992778 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5058373}
W20250623 08:04:43.994088 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 334) could send its response
I20250623 08:04:43.994408 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 335) took 5059 ms. Trace:
I20250623 08:04:43.994463 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_f1d8f7ef7cf144eaa1fd741b10eac3d6 [id=9060c1640fca461c9020722b3165d144]
I20250623 08:04:43.994791 981 rpcz_store.cc:276] 0623 08:04:38.934569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.934791 (+ 222us) service_pool.cc:225] Handling call
0623 08:04:43.994336 (+5059545us) ts_tablet_manager.cc:1145] Deleting tablet 358cb6ac24414bdfbc32155e732f9ac8
0623 08:04:43.994345 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.994395 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5059380}
I20250623 08:04:43.994719 343 raft_consensus.cc:383] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:43.995209 343 raft_consensus.cc:738] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:43.995507 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 335) could send its response
W20250623 08:04:43.995827 32453 leader_election.cc:343] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.995870 981 tablet_replica.cc:331] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:43.996090 2160 consensus_queue.cc:1035] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.996201 32515 raft_consensus.cc:2466] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
W20250623 08:04:43.996600 32453 leader_election.cc:343] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:43.996524 343 consensus_queue.cc:260] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:43.997033 981 raft_consensus.cc:2241] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:43.996701 32453 leader_election.cc:304] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:43.997876 2160 raft_consensus.cc:2747] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:43.997920 981 raft_consensus.cc:2270] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:43.998322 2160 consensus_queue.cc:1035] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:43.998731 343 ts_tablet_manager.cc:1428] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:43.999409 981 ts_tablet_manager.cc:1905] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:43.999637 2160 raft_consensus.cc:491] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.000027 343 tablet_bootstrap.cc:492] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.001202 2160 raft_consensus.cc:513] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.001752 2160 leader_election.cc:290] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.002048 32589 leader_election.cc:304] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.002344 2080 raft_consensus.cc:2802] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.002470 2080 raft_consensus.cc:695] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.002686 2080 consensus_queue.cc:237] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.004801 343 tablet_bootstrap.cc:654] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.012553 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_cc3b284f368e4123bf1a2da98f4f4fa8 [id=906debc01a6f45a49a86b5f939ac1dee]
I20250623 08:04:44.015508 2014 consensus_queue.cc:1035] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.018662 343 tablet_bootstrap.cc:492] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.019659 342 tablet_bootstrap.cc:492] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.019332 2013 raft_consensus.cc:491] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.019960 2013 raft_consensus.cc:513] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.019966 343 ts_tablet_manager.cc:1397] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20250623 08:04:44.020051 342 ts_tablet_manager.cc:1397] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20250623 08:04:44.020582 2013 leader_election.cc:290] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.020947 343 raft_consensus.cc:357] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.021968 343 raft_consensus.cc:383] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.021852 342 raft_consensus.cc:357] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.022162 343 raft_consensus.cc:738] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.022322 343 consensus_queue.cc:260] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.022333 342 raft_consensus.cc:383] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.022514 342 raft_consensus.cc:738] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.022964 342 consensus_queue.cc:260] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.023185 343 ts_tablet_manager.cc:1428] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250623 08:04:44.024415 981 ts_tablet_manager.cc:1918] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:44.025090 343 tablet_bootstrap.cc:492] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.025810 342 ts_tablet_manager.cc:1428] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250623 08:04:44.025660 339 tablet_bootstrap.cc:492] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.026763 339 ts_tablet_manager.cc:1397] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.080s user 0.014s sys 0.006s
I20250623 08:04:44.027488 339 raft_consensus.cc:357] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.027841 339 raft_consensus.cc:383] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.027942 339 raft_consensus.cc:738] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.028183 339 consensus_queue.cc:260] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.028074 342 tablet_bootstrap.cc:492] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:44.028656 339 ts_tablet_manager.cc:1428] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:44.028942 339 tablet_bootstrap.cc:492] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.030526 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db81250d1e164a7d94affd764fc0ff0c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.030736 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1ef88ab1f1cd49139106aed3732b50a8 [id=90865844914743379da60c53e1c3a0d5]
I20250623 08:04:44.032534 343 tablet_bootstrap.cc:654] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.031697 339 tablet_bootstrap.cc:654] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:44.039829 32589 leader_election.cc:343] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.031772 981 log.cc:1199] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/3249605158634f4eab5b9d61357c383f
I20250623 08:04:44.040292 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_39a59f2375e743109b067b2de8b6d4db [id=9125a8d179614084bebc3591092905e5]
I20250623 08:04:44.040494 981 ts_tablet_manager.cc:1939] T 3249605158634f4eab5b9d61357c383f P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.040829 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_31ab57df769c4b2cb4e2cb74d69b4eb5 [id=92602c2df9c44292abbed7d6095922c9]
I20250623 08:04:44.041254 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 336) took 5093 ms. Trace:
I20250623 08:04:44.041337 981 rpcz_store.cc:276] 0623 08:04:38.947687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.947747 (+ 60us) service_pool.cc:225] Handling call
0623 08:04:43.995825 (+5048078us) ts_tablet_manager.cc:1145] Deleting tablet 3249605158634f4eab5b9d61357c383f
0623 08:04:43.995836 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.008138 (+ 12302us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.024315 (+ 16177us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.041235 (+ 16920us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5047873}
I20250623 08:04:44.041494 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_e69f9fec5b604bca8658b2124145f92e [id=96cf904f9ea64544ae6f662208874f89]
W20250623 08:04:44.041559 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 336) could send its response
I20250623 08:04:44.041698 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 337) took 5087 ms. Trace:
I20250623 08:04:44.041754 981 rpcz_store.cc:276] 0623 08:04:38.954190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.954244 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:44.041599 (+5087355us) ts_tablet_manager.cc:1145] Deleting tablet a25bdc9b7e48467fba3771365e58bcbc
0623 08:04:44.041609 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.041686 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5087125}
W20250623 08:04:44.041920 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 337) could send its response
I20250623 08:04:44.042022 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 338) took 5073 ms. Trace:
I20250623 08:04:44.042021 2014 raft_consensus.cc:491] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.042075 981 rpcz_store.cc:276] 0623 08:04:38.968705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.968766 (+ 61us) service_pool.cc:225] Handling call
0623 08:04:44.041952 (+5073186us) ts_tablet_manager.cc:1145] Deleting tablet a9c8d480b8ba49a1be140b0e0a79f4e4
0623 08:04:44.041959 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.042011 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5072978}
I20250623 08:04:44.042182 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_68f032ade7314d0da985fa8c94b11a8c [id=9895a12eef5a415cacd19cd277721b70]
W20250623 08:04:44.042232 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 338) could send its response
I20250623 08:04:44.042158 2014 raft_consensus.cc:513] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.042326 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 339) took 5070 ms. Trace:
I20250623 08:04:44.042378 981 rpcz_store.cc:276] 0623 08:04:38.972246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.972302 (+ 56us) service_pool.cc:225] Handling call
0623 08:04:44.042260 (+5069958us) ts_tablet_manager.cc:1145] Deleting tablet 485538f7c63948818f20a25c02381519
0623 08:04:44.042268 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.042315 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5069750}
W20250623 08:04:44.042558 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 339) could send its response
I20250623 08:04:44.042650 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 340) took 5068 ms. Trace:
I20250623 08:04:44.033020 1682 raft_consensus.cc:491] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.042825 1682 raft_consensus.cc:513] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.042944 2014 leader_election.cc:290] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.043344 1682 leader_election.cc:290] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.043426 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.043766 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "803d3607d75443568b7ddc9f0c0bdc90" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:44.043897 32589 leader_election.cc:343] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.042722 981 rpcz_store.cc:276] 0623 08:04:38.974569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.974611 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:44.042586 (+5067975us) ts_tablet_manager.cc:1145] Deleting tablet effd4fff43fe4d4fb36458986e1dfce1
0623 08:04:44.042593 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.042639 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5067770}
I20250623 08:04:44.044138 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 342) took 5043 ms. Trace:
I20250623 08:04:44.044205 981 rpcz_store.cc:276] 0623 08:04:39.000955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.001003 (+ 48us) service_pool.cc:225] Handling call
0623 08:04:44.044067 (+5043064us) ts_tablet_manager.cc:1145] Deleting tablet ea702abb31fb41498ab61bbb05b1f7e2
0623 08:04:44.044076 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.044124 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5042874}
W20250623 08:04:44.044528 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 340) could send its response
W20250623 08:04:44.044687 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 342) could send its response
I20250623 08:04:44.045305 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "803d3607d75443568b7ddc9f0c0bdc90" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.046000 32314 leader_election.cc:304] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:44.046296 1682 raft_consensus.cc:2747] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.044507 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 343) took 5042 ms. Trace:
I20250623 08:04:44.046861 981 rpcz_store.cc:276] 0623 08:04:39.002146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.002188 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:44.044442 (+5042254us) ts_tablet_manager.cc:1145] Deleting tablet 4cb66c7a273a412db3ae77af9e30d036
0623 08:04:44.044449 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.044493 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5042124}
W20250623 08:04:44.047077 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 343) could send its response
I20250623 08:04:44.047362 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 344) took 5039 ms. Trace:
I20250623 08:04:44.047463 981 rpcz_store.cc:276] 0623 08:04:39.008196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.008240 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:44.047267 (+5039027us) ts_tablet_manager.cc:1145] Deleting tablet 2ecf4af9e476495683bd63cac20d4672
0623 08:04:44.047279 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.047346 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4982812}
W20250623 08:04:44.047670 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 344) could send its response
I20250623 08:04:44.047927 981 tablet_replica.cc:331] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:44.048156 981 raft_consensus.cc:2241] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.048342 981 raft_consensus.cc:2270] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.049050 981 ts_tablet_manager.cc:1905] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.048995 952 ts_tablet_manager.cc:1918] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:44.050156 952 log.cc:1199] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a8945b6f49474648ac73bcb85285680a
I20250623 08:04:44.054224 952 ts_tablet_manager.cc:1939] T a8945b6f49474648ac73bcb85285680a P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:44.057020 343 tablet_bootstrap.cc:492] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.056865 342 tablet_bootstrap.cc:654] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.057614 343 ts_tablet_manager.cc:1397] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.008s
I20250623 08:04:44.059165 32380 raft_consensus.cc:3058] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.059108 343 raft_consensus.cc:357] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.059782 343 raft_consensus.cc:383] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.059991 343 raft_consensus.cc:738] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.060765 339 tablet_bootstrap.cc:492] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.060926 339 ts_tablet_manager.cc:1397] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.003s
I20250623 08:04:44.061527 339 raft_consensus.cc:357] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.061731 339 raft_consensus.cc:383] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.061815 339 raft_consensus.cc:738] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.062080 339 consensus_queue.cc:260] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.063717 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c3dcdcaa2e14cb2bd02ebb8f1d0e159" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.067102 339 ts_tablet_manager.cc:1428] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:44.067381 339 tablet_bootstrap.cc:492] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.068202 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 336) took 4736 ms. Trace:
I20250623 08:04:44.068296 952 rpcz_store.cc:276] 0623 08:04:39.331680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.333139 (+ 1459us) service_pool.cc:225] Handling call
0623 08:04:43.975187 (+4642048us) ts_tablet_manager.cc:1145] Deleting tablet a8945b6f49474648ac73bcb85285680a
0623 08:04:43.975199 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:43.997932 (+ 22733us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.047963 (+ 50031us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.068180 (+ 20217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4519,"tablet-delete.queue_time_us":4641518}
I20250623 08:04:44.068667 952 tablet_replica.cc:331] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:44.068869 952 raft_consensus.cc:2241] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.069036 952 raft_consensus.cc:2270] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.073324 981 ts_tablet_manager.cc:1918] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:44.073428 981 log.cc:1199] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/bfd3e263464c4f2c8af9262acf211193
I20250623 08:04:44.073794 981 ts_tablet_manager.cc:1939] T bfd3e263464c4f2c8af9262acf211193 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.074479 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 345) took 5065 ms. Trace:
I20250623 08:04:44.074555 981 rpcz_store.cc:276] 0623 08:04:39.009209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.012581 (+ 3372us) service_pool.cc:225] Handling call
0623 08:04:44.047867 (+5035286us) ts_tablet_manager.cc:1145] Deleting tablet bfd3e263464c4f2c8af9262acf211193
0623 08:04:44.047879 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.056938 (+ 9059us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.073220 (+ 16282us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.074463 (+ 1243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4983348}
I20250623 08:04:44.075002 343 consensus_queue.cc:260] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.064199 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c3dcdcaa2e14cb2bd02ebb8f1d0e159" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.075400 32654 raft_consensus.cc:2466] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:44.075523 343 ts_tablet_manager.cc:1428] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20250623 08:04:44.075811 343 tablet_bootstrap.cc:492] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.064574 2014 consensus_queue.cc:1035] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.076311 1627 raft_consensus.cc:491] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.076455 1627 raft_consensus.cc:513] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.077013 1627 leader_election.cc:290] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.077461 343 tablet_bootstrap.cc:654] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:44.078361 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 345) could send its response
I20250623 08:04:44.078482 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 346) took 5068 ms. Trace:
I20250623 08:04:44.078560 981 rpcz_store.cc:276] 0623 08:04:39.010035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.012746 (+ 2711us) service_pool.cc:225] Handling call
0623 08:04:44.078384 (+5065638us) ts_tablet_manager.cc:1145] Deleting tablet 965f1f0a107b4653b6aac01e0d8e259a
0623 08:04:44.078394 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.078465 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5013811}
I20250623 08:04:44.078920 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 348) took 5058 ms. Trace:
I20250623 08:04:44.079059 981 rpcz_store.cc:276] 0623 08:04:39.020730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.020786 (+ 56us) service_pool.cc:225] Handling call
0623 08:04:44.078840 (+5058054us) ts_tablet_manager.cc:1145] Deleting tablet 38ecd4c243b840f4b66d8e01078dca64
0623 08:04:44.078848 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.078907 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5012949}
W20250623 08:04:44.079255 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 346) could send its response
W20250623 08:04:44.079522 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 348) could send its response
I20250623 08:04:44.079536 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 347) took 5060 ms. Trace:
I20250623 08:04:44.081240 981 rpcz_store.cc:276] 0623 08:04:39.018858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.023081 (+ 4223us) service_pool.cc:225] Handling call
0623 08:04:44.079330 (+5056249us) ts_tablet_manager.cc:1145] Deleting tablet f3d32798168345edb9898751d083057c
0623 08:04:44.079458 (+ 128us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.079522 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5013384}
W20250623 08:04:44.081504 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 347) could send its response
I20250623 08:04:44.081821 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 349) took 5058 ms. Trace:
I20250623 08:04:44.081923 981 rpcz_store.cc:276] 0623 08:04:39.023818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.023866 (+ 48us) service_pool.cc:225] Handling call
0623 08:04:44.081730 (+5057864us) ts_tablet_manager.cc:1145] Deleting tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0
0623 08:04:44.081742 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.081806 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5015743}
W20250623 08:04:44.082139 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 349) could send its response
I20250623 08:04:44.082451 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 350) took 5056 ms. Trace:
I20250623 08:04:44.082558 981 rpcz_store.cc:276] 0623 08:04:39.025577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.025627 (+ 50us) service_pool.cc:225] Handling call
0623 08:04:44.082341 (+5056714us) ts_tablet_manager.cc:1145] Deleting tablet f279b63b55c64e548ded27b02025c9f5
0623 08:04:44.082351 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.082438 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5016305}
W20250623 08:04:44.082815 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 350) could send its response
I20250623 08:04:44.083070 981 tablet_replica.cc:331] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:44.083292 981 raft_consensus.cc:2241] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.083506 981 raft_consensus.cc:2270] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.084106 981 ts_tablet_manager.cc:1905] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.088886 342 tablet_bootstrap.cc:492] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.089193 342 ts_tablet_manager.cc:1397] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.000s
W20250623 08:04:44.091617 32453 leader_election.cc:343] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:44.092036 32589 consensus_peers.cc:487] T a0e9b2f810014e81a29ca9b41cd3f8ba P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:44.092702 2160 consensus_queue.cc:1035] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 08:04:44.093533 952 ts_tablet_manager.cc:1905] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.095222 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db81250d1e164a7d94affd764fc0ff0c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.095319 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.095389 32514 raft_consensus.cc:2466] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.095476 32519 raft_consensus.cc:2466] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.098574 342 raft_consensus.cc:357] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.098762 342 raft_consensus.cc:383] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.098829 342 raft_consensus.cc:738] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.099018 342 consensus_queue.cc:260] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.099884 342 ts_tablet_manager.cc:1428] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250623 08:04:44.100332 342 tablet_bootstrap.cc:492] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:44.100903 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 336) could send its response
I20250623 08:04:44.103263 342 tablet_bootstrap.cc:654] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.104807 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_49e1ee7e0d024625988be24a823a443c [id=9a7a76e61e1b43508b6f6080f5163417]
I20250623 08:04:44.105070 2014 raft_consensus.cc:491] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.105330 2014 raft_consensus.cc:513] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.106079 2014 leader_election.cc:290] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.106325 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_dd2b39480694444ba170c65257a8dc58 [id=9dd60e056f8b4c17b9d8d9569ebd3447]
I20250623 08:04:44.107311 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1911a70d3e224ae4b9805e345c99624b [id=9f504d8065a24602af6e2142069438a5]
I20250623 08:04:44.110843 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9891997e026448e39943ae13082bfe03" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:44.111526 32589 leader_election.cc:343] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.113600 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.114316 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.114562 32646 raft_consensus.cc:2466] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:44.119453 32020 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskJxinPB/build/asan/bin/kudu
/tmp/dist-test-taskJxinPB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.62:39465
--webserver_interface=127.31.69.62
--webserver_port=33855
--builtin_ntp_servers=127.31.69.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250623 08:04:44.124526 32589 leader_election.cc:304] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.124913 2014 raft_consensus.cc:2802] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.125061 2014 raft_consensus.cc:491] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.125191 2014 raft_consensus.cc:3058] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.126577 952 ts_tablet_manager.cc:1918] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:44.128479 2014 raft_consensus.cc:513] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.128526 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_7ccb5edd6b824f308d349c51d926d963 [id=9f97f6deca194137a353ea221d0e3676]
I20250623 08:04:44.129251 2014 leader_election.cc:290] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.129884 32589 leader_election.cc:304] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.130254 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]
I20250623 08:04:44.130182 2014 consensus_queue.cc:1035] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.130975 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4]
I20250623 08:04:44.131182 2188 raft_consensus.cc:2802] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.131269 2188 raft_consensus.cc:491] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.131366 2188 raft_consensus.cc:3058] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.134676 2188 raft_consensus.cc:513] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.135380 2188 leader_election.cc:290] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.136610 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db81250d1e164a7d94affd764fc0ff0c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.139672 981 ts_tablet_manager.cc:1918] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:44.139895 981 log.cc:1199] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b80a91ae2c084020b5d3444b570b28f2
W20250623 08:04:44.144526 32589 leader_election.cc:343] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.146898 952 log.cc:1199] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f4a96ae9190640dea11bd11a9f0e4372
I20250623 08:04:44.147671 952 ts_tablet_manager.cc:1939] T f4a96ae9190640dea11bd11a9f0e4372 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:44.137934 32453 leader_election.cc:304] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.149187 1627 raft_consensus.cc:2802] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.131783 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]
I20250623 08:04:44.140215 339 tablet_bootstrap.cc:654] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.153321 981 ts_tablet_manager.cc:1939] T b80a91ae2c084020b5d3444b570b28f2 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.144115 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.149585 32453 leader_election.cc:304] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:44.154309 1957 raft_consensus.cc:2802] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.154413 1957 raft_consensus.cc:491] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.154510 1957 raft_consensus.cc:3058] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.157784 1359 consensus_queue.cc:1035] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.158407 1957 raft_consensus.cc:513] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.158969 32380 raft_consensus.cc:3058] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.159564 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9891997e026448e39943ae13082bfe03" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.159601 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db81250d1e164a7d94affd764fc0ff0c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:44.159713 32518 raft_consensus.cc:2466] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.159754 32519 raft_consensus.cc:3058] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.160023 1957 leader_election.cc:290] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.160347 1627 raft_consensus.cc:491] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.160552 1627 raft_consensus.cc:3058] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.162209 32519 raft_consensus.cc:2466] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:44.150249 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 338) took 4803 ms. Trace:
I20250623 08:04:44.162773 952 rpcz_store.cc:276] 0623 08:04:39.346406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.346459 (+ 53us) service_pool.cc:225] Handling call
0623 08:04:44.068615 (+4722156us) ts_tablet_manager.cc:1145] Deleting tablet f4a96ae9190640dea11bd11a9f0e4372
0623 08:04:44.068626 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.123542 (+ 54916us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.126482 (+ 2940us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.148899 (+ 22417us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23919,"tablet-delete.queue_time_us":4721973}
I20250623 08:04:44.167332 1627 raft_consensus.cc:513] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.169749 1627 leader_election.cc:290] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.171847 343 tablet_bootstrap.cc:492] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.172922 339 tablet_bootstrap.cc:492] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.173108 339 ts_tablet_manager.cc:1397] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.106s user 0.001s sys 0.012s
I20250623 08:04:44.178366 342 tablet_bootstrap.cc:492] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.178514 342 ts_tablet_manager.cc:1397] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.000s
I20250623 08:04:44.173790 339 raft_consensus.cc:357] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.180002 339 raft_consensus.cc:383] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.180114 339 raft_consensus.cc:738] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.180364 339 consensus_queue.cc:260] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.181064 339 ts_tablet_manager.cc:1428] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250623 08:04:44.164175 952 tablet_replica.cc:331] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:44.181352 339 tablet_bootstrap.cc:492] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.181679 952 raft_consensus.cc:2241] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.182951 339 tablet_bootstrap.cc:654] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:44.176899 32589 leader_election.cc:343] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.177071 2014 raft_consensus.cc:491] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.183218 2014 raft_consensus.cc:513] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.182089 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:44.183804 32518 raft_consensus.cc:2391] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:44.183917 2014 leader_election.cc:290] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.184607 1823 consensus_queue.cc:1035] T 3c14e325fa5f4f848748c346bc25dde0 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:44.185767 32453 leader_election.cc:343] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.186944 952 raft_consensus.cc:2270] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.187530 952 ts_tablet_manager.cc:1905] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.187992 2188 raft_consensus.cc:491] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.188148 2188 raft_consensus.cc:513] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.188776 2188 leader_election.cc:290] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.187798 342 raft_consensus.cc:357] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.189495 342 raft_consensus.cc:383] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.189903 1359 consensus_queue.cc:1035] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.190512 342 raft_consensus.cc:738] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.193550 343 ts_tablet_manager.cc:1397] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.118s user 0.026s sys 0.016s
I20250623 08:04:44.194356 343 raft_consensus.cc:357] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:44.195822 32589 consensus_peers.cc:487] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:44.183084 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 353) took 5150 ms. Trace:
I20250623 08:04:44.196245 343 raft_consensus.cc:383] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.196331 343 raft_consensus.cc:738] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.186013 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90ceb0f8d5c94c4e9427042d8a35e16c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.189217 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bfa787f828b411ab68d81bb896f7fb6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.189791 1682 raft_consensus.cc:491] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:44.196887 1682 raft_consensus.cc:513] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.197521 1682 leader_election.cc:290] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:44.197860 32589 leader_election.cc:343] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.197928 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa398354a9e44633857a75c0bbba08b3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:44.198129 32589 leader_election.cc:343] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.198273 342 consensus_queue.cc:260] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:44.198634 32314 leader_election.cc:343] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: aa398354a9e44633857a75c0bbba08b3
I20250623 08:04:44.198791 342 ts_tablet_manager.cc:1428] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20250623 08:04:44.198796 32314 leader_election.cc:304] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:44.199054 342 tablet_bootstrap.cc:492] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:44.199098 1682 raft_consensus.cc:2747] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:44.200663 342 tablet_bootstrap.cc:654] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.196889 981 rpcz_store.cc:276] 0623 08:04:39.032926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.032980 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:44.083024 (+5050044us) ts_tablet_manager.cc:1145] Deleting tablet b80a91ae2c084020b5d3444b570b28f2
0623 08:04:44.083035 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.121253 (+ 38218us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.139495 (+ 18242us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.183064 (+ 43569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5016014}
I20250623 08:04:44.201412 981 tablet_replica.cc:331] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
W20250623 08:04:44.201375 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 353) could send its response
I20250623 08:04:44.201601 981 raft_consensus.cc:2241] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.201750 981 raft_consensus.cc:2270] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.202234 981 ts_tablet_manager.cc:1905] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.202546 343 consensus_queue.cc:260] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.204831 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_71f2f0414e0840be832096955576cb2f [id=a6918c5eb39a435ab4b035c1509916a0]
I20250623 08:04:44.205684 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_973ac8c9b50b4fb2a8d83318384a8b76 [id=ab80a8647b644e6ebbdfba0d2e76107a]
I20250623 08:04:44.206461 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_f0683327c2bf468582a2f3e557d15725 [id=acb1a73cc8b546dab7431a57c3a8857e]
I20250623 08:04:44.207787 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_f66b3fa5edc14cffaa95a0a0a4f62f67 [id=ad4cb54e99034dd3af8bdac8fcce4669]
I20250623 08:04:44.208487 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_b764664d1bdc4441907690b6616f3570 [id=ad7210d2665f463c8591e79cf62d15dc]
I20250623 08:04:44.212266 343 ts_tablet_manager.cc:1428] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250623 08:04:44.212662 343 tablet_bootstrap.cc:492] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.214382 343 tablet_bootstrap.cc:654] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.215613 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]
I20250623 08:04:44.219425 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:44.220983 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 338) could send its response
I20250623 08:04:44.222012 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c3dcdcaa2e14cb2bd02ebb8f1d0e159" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.218508 32589 leader_election.cc:304] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.222970 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c3dcdcaa2e14cb2bd02ebb8f1d0e159" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:44.223171 32654 raft_consensus.cc:3058] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.226320 32654 raft_consensus.cc:2466] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:44.220460 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:44.226711 32647 raft_consensus.cc:2391] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:44.226814 32589 leader_election.cc:304] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.227311 32589 leader_election.cc:304] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.231307 2013 raft_consensus.cc:2802] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.231603 2014 consensus_queue.cc:1035] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.233546 2080 raft_consensus.cc:2747] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.223168 2188 raft_consensus.cc:2802] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.234714 2188 raft_consensus.cc:491] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.234834 2188 raft_consensus.cc:3058] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.235546 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_f883c19d25584eb185a1e31ef6d88d4f [id=ae1d0fae75be48768a6503498a33df32]
I20250623 08:04:44.236358 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1476eae1111b4f1dbd7c0f5dd0dd9b34 [id=b005275508b748c7a8494ffebdc0bd49]
I20250623 08:04:44.237121 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00]
I20250623 08:04:44.237857 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_da6195995899479b879a3d41e0c07ed4 [id=b3b1a699a6b44ff09cf7b4f105e04794]
I20250623 08:04:44.237888 2188 raft_consensus.cc:513] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.238308 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_905d0330c27149a19dabb586c51a2e0a [id=b3f407a7a26e4bd5ae9a719a5c4577c0]
I20250623 08:04:44.238507 2188 leader_election.cc:290] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.240883 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9891997e026448e39943ae13082bfe03" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:44.241708 32589 leader_election.cc:343] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.242844 981 ts_tablet_manager.cc:1918] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:44.245667 981 log.cc:1199] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/dc0dad440b494018bc0d6cfd40940bc1
I20250623 08:04:44.247758 981 ts_tablet_manager.cc:1939] T dc0dad440b494018bc0d6cfd40940bc1 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.249119 2188 raft_consensus.cc:491] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.249365 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 351) took 5219 ms. Trace:
I20250623 08:04:44.249395 2188 raft_consensus.cc:513] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.250146 2188 leader_election.cc:290] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.250433 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11bb55088ab24b94af3e509810890247" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.253995 981 rpcz_store.cc:276] 0623 08:04:39.029571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.033113 (+ 3542us) service_pool.cc:225] Handling call
0623 08:04:44.201365 (+5168252us) ts_tablet_manager.cc:1145] Deleting tablet dc0dad440b494018bc0d6cfd40940bc1
0623 08:04:44.201375 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.216991 (+ 15616us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.242709 (+ 25718us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.249347 (+ 6638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5134297}
W20250623 08:04:44.254907 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 351) could send its response
I20250623 08:04:44.256757 339 tablet_bootstrap.cc:492] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.256935 339 ts_tablet_manager.cc:1397] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.007s
W20250623 08:04:44.257925 32453 consensus_peers.cc:487] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9ea74d20c5f4eefbcb8508ed1774026. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:44.258328 32453 consensus_peers.cc:487] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1be8f8557b9a4ae2917c37b8d01cedf6. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:44.258960 1823 consensus_queue.cc:1035] T 8e17e1ca404b462da43aa1db7dc3b556 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.259228 32453 leader_election.cc:304] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:44.259666 1953 raft_consensus.cc:2802] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.259775 1953 raft_consensus.cc:695] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:44.259976 1953 consensus_queue.cc:237] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.266851 952 ts_tablet_manager.cc:1918] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250623 08:04:44.267768 339 raft_consensus.cc:357] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.267974 339 raft_consensus.cc:383] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.268030 339 raft_consensus.cc:738] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:44.268222 32589 leader_election.cc:343] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.268988 2013 raft_consensus.cc:695] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.269456 2013 consensus_queue.cc:237] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.270934 339 consensus_queue.cc:260] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.271639 339 ts_tablet_manager.cc:1428] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250623 08:04:44.272030 952 log.cc:1199] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/ba3d2fa01d574dd992dc904ac024ba3a
I20250623 08:04:44.272498 952 ts_tablet_manager.cc:1939] T ba3d2fa01d574dd992dc904ac024ba3a P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:44.272720 343 tablet_bootstrap.cc:492] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.272922 343 ts_tablet_manager.cc:1397] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.060s user 0.000s sys 0.009s
I20250623 08:04:44.254976 981 tablet_replica.cc:331] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:44.273061 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_a50378b31cda420a8791443844eed5f2 [id=b83148d602ab4c19a5fb1b4072b626e0]
I20250623 08:04:44.273756 343 raft_consensus.cc:357] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.273837 339 tablet_bootstrap.cc:492] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.273916 343 raft_consensus.cc:383] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.273979 343 raft_consensus.cc:738] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.275949 339 tablet_bootstrap.cc:654] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.277762 981 raft_consensus.cc:2241] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.283841 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 339) took 4937 ms. Trace:
I20250623 08:04:44.284003 952 rpcz_store.cc:276] 0623 08:04:39.346896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.346943 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:44.163378 (+4816435us) ts_tablet_manager.cc:1145] Deleting tablet ba3d2fa01d574dd992dc904ac024ba3a
0623 08:04:44.163389 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.239355 (+ 75966us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.266687 (+ 27332us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.283812 (+ 17125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3109,"tablet-delete.queue_time_us":4816169}
I20250623 08:04:44.284379 952 tablet_replica.cc:331] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:44.284962 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_5338d405e4ed40589c691093043355d1 [id=b9246ed2ad6242739ae14a362b4af828]
I20250623 08:04:44.285331 981 pending_rounds.cc:70] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:44.285516 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1671bbefe7bd4d4b86a917373288a1a6 [id=baaeae53fdcb49b7889d8379fa203c79]
I20250623 08:04:44.285708 981 pending_rounds.cc:77] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727426929065984 op_type: NO_OP noop_request { }
I20250623 08:04:44.285992 981 raft_consensus.cc:2887] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:44.286245 981 raft_consensus.cc:2270] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.286398 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_412c3eb097714d94bf0b6c487f1c3caa [id=bbf01747113a4685a0cd746091a9e5bc]
I20250623 08:04:44.287315 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]
I20250623 08:04:44.286902 342 tablet_bootstrap.cc:492] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.287942 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_977fe8fc665f4267b394065e22f67eea [id=be9754ecb40a413d9940224adbd7e2eb]
I20250623 08:04:44.287973 342 ts_tablet_manager.cc:1397] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.000s
I20250623 08:04:44.288372 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]
I20250623 08:04:44.288756 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_0666030ad7ec4ef89335cdbaf0cca443 [id=bf6ca5aa8d5044868244a9b37cfa9519]
I20250623 08:04:44.289152 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa]
I20250623 08:04:44.290614 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90ceb0f8d5c94c4e9427042d8a35e16c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.290817 32518 raft_consensus.cc:2466] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.291051 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bfa787f828b411ab68d81bb896f7fb6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.291220 32515 raft_consensus.cc:2466] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.291338 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11bb55088ab24b94af3e509810890247" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.291438 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9891997e026448e39943ae13082bfe03" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:44.291476 32514 raft_consensus.cc:2466] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.291553 32513 raft_consensus.cc:3058] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.292212 342 raft_consensus.cc:357] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.292505 981 ts_tablet_manager.cc:1905] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.295497 1359 raft_consensus.cc:491] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.295770 32513 raft_consensus.cc:2466] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:44.308924 952 raft_consensus.cc:2241] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:44.308979 1359 raft_consensus.cc:513] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.309479 952 pending_rounds.cc:70] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:44.309581 952 pending_rounds.cc:77] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727461933142016 op_type: NO_OP noop_request { }
I20250623 08:04:44.309654 1359 leader_election.cc:290] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.310550 339 tablet_bootstrap.cc:492] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.310741 339 ts_tablet_manager.cc:1397] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20250623 08:04:44.308333 342 raft_consensus.cc:383] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.311599 339 raft_consensus.cc:357] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.311658 342 raft_consensus.cc:738] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.311902 339 raft_consensus.cc:383] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.312016 339 raft_consensus.cc:738] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.312286 339 consensus_queue.cc:260] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.312289 342 consensus_queue.cc:260] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.312856 339 ts_tablet_manager.cc:1428] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:44.312996 342 ts_tablet_manager.cc:1428] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250623 08:04:44.313235 339 tablet_bootstrap.cc:492] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.313460 342 tablet_bootstrap.cc:492] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:44.314993 339 tablet_bootstrap.cc:654] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.315198 342 tablet_bootstrap.cc:654] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.309710 952 raft_consensus.cc:2887] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:44.316051 952 raft_consensus.cc:2270] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.317044 952 ts_tablet_manager.cc:1905] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.322821 339 tablet_bootstrap.cc:492] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.322965 339 ts_tablet_manager.cc:1397] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250623 08:04:44.325367 339 raft_consensus.cc:357] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.325547 339 raft_consensus.cc:383] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.325596 339 raft_consensus.cc:738] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.325809 339 consensus_queue.cc:260] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:44.329587 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 339) could send its response
I20250623 08:04:44.332146 2210 consensus_queue.cc:1035] T 5443f12974bf41929d9cd937f146cd56 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.333389 339 ts_tablet_manager.cc:1428] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250623 08:04:44.334450 339 tablet_bootstrap.cc:492] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:44.334941 32589 consensus_peers.cc:487] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:44.336278 32589 leader_election.cc:304] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.336700 2210 raft_consensus.cc:2802] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.336772 2210 raft_consensus.cc:491] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.336825 2210 raft_consensus.cc:3058] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.339982 339 tablet_bootstrap.cc:654] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.340729 2210 raft_consensus.cc:513] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.341768 2210 leader_election.cc:290] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.342048 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90ceb0f8d5c94c4e9427042d8a35e16c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:44.342577 32589 leader_election.cc:343] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.343367 32589 leader_election.cc:304] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.343770 2210 raft_consensus.cc:2802] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.343868 2210 raft_consensus.cc:491] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.344079 2210 raft_consensus.cc:3058] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.346736 2210 raft_consensus.cc:513] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.347645 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bfa787f828b411ab68d81bb896f7fb6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:44.348134 32589 leader_election.cc:343] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.348567 2210 leader_election.cc:290] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:44.348951 32314 consensus_peers.cc:487] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba3d2fa01d574dd992dc904ac024ba3a. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:44.349520 32589 leader_election.cc:304] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.349936 2210 raft_consensus.cc:2802] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.350034 2210 raft_consensus.cc:491] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.350136 2210 raft_consensus.cc:3058] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.352664 2210 raft_consensus.cc:513] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.353291 343 consensus_queue.cc:260] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.355728 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]
W20250623 08:04:44.359526 32453 leader_election.cc:343] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.360038 32453 leader_election.cc:304] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.361101 1359 raft_consensus.cc:2747] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.362083 952 ts_tablet_manager.cc:1918] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:44.362210 952 log.cc:1199] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/6c3dcdcaa2e14cb2bd02ebb8f1d0e159
I20250623 08:04:44.362608 952 ts_tablet_manager.cc:1939] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:44.363118 339 tablet_bootstrap.cc:492] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.363310 339 ts_tablet_manager.cc:1397] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250623 08:04:44.363384 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11bb55088ab24b94af3e509810890247" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.363639 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_d2473e1e63c84b229243a1c676266ca6 [id=c63ada5e02d74e049460990450ab4962]
I20250623 08:04:44.364063 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]
I20250623 08:04:44.364156 339 raft_consensus.cc:357] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.364470 339 raft_consensus.cc:383] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.364487 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_b161fc2726dc4831988f5ac03028b089 [id=c8effd1e07264a5984e51f380a700652]
I20250623 08:04:44.364598 339 raft_consensus.cc:738] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.364876 339 consensus_queue.cc:260] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:44.362783 32453 leader_election.cc:343] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.365413 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517]
I20250623 08:04:44.365864 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]
I20250623 08:04:44.366461 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_ff6e3cec929a433b8ac68152b9095dfc [id=ce7434eee7664a8686eb22f28df7f1d8]
I20250623 08:04:44.353735 2013 consensus_queue.cc:1035] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:44.364071 32589 leader_election.cc:343] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.368065 1682 raft_consensus.cc:491] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.368173 1682 raft_consensus.cc:513] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.368711 1682 leader_election.cc:290] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.368901 1682 raft_consensus.cc:491] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.368991 1682 raft_consensus.cc:513] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.369191 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 347) took 5010 ms. Trace:
I20250623 08:04:44.369848 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16331ba6a34942a5a1b4ff4ac7bac55e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.369848 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74da35cd34c14284af004001b25aa531" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.370090 32513 raft_consensus.cc:2391] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:44.369268 952 rpcz_store.cc:276] 0623 08:04:39.358289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.358648 (+ 359us) service_pool.cc:225] Handling call
0623 08:04:44.284332 (+4925684us) ts_tablet_manager.cc:1145] Deleting tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159
0623 08:04:44.284340 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.309314 (+ 24974us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x6160002d0a54. stack: 00007f21fcee4d56 00007f21fcee4b10 00007f21fbedb29d 0000000000970a84 0000000000970a4d 000000000097041a 00007f2208942e80 00007f2209c54a93 00007f220c730c97 00007f220c762d24 00007f220c744780 00007f220c74450d 0000000000a751bc 00007f21fcf2a3ff 00007f21fcf2d6cb 00007f21fcf2d548
0623 08:04:44.320418 (+ 11104us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.361956 (+ 41538us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.369170 (+ 7214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55992192,"tablet-delete.queue_time_us":4925430}
I20250623 08:04:44.370570 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 349) took 5012 ms. Trace:
I20250623 08:04:44.370664 952 rpcz_store.cc:276] 0623 08:04:39.358629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.358706 (+ 77us) service_pool.cc:225] Handling call
0623 08:04:44.370488 (+5011782us) ts_tablet_manager.cc:1145] Deleting tablet a9c8d480b8ba49a1be140b0e0a79f4e4
0623 08:04:44.370497 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.370556 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5011563}
I20250623 08:04:44.370887 1682 leader_election.cc:290] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.371004 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 346) took 5012 ms. Trace:
I20250623 08:04:44.371093 952 rpcz_store.cc:276] 0623 08:04:39.358129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.359029 (+ 900us) service_pool.cc:225] Handling call
0623 08:04:44.370890 (+5011861us) ts_tablet_manager.cc:1145] Deleting tablet 1b213c8c685c43868df1e814c79fc61f
0623 08:04:44.370900 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.370975 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5011714}
I20250623 08:04:44.371340 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 350) took 5012 ms. Trace:
I20250623 08:04:44.371407 952 rpcz_store.cc:276] 0623 08:04:39.359118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.359188 (+ 70us) service_pool.cc:225] Handling call
0623 08:04:44.371252 (+5012064us) ts_tablet_manager.cc:1145] Deleting tablet 1a070895f4984493a3b9f6fa779797a9
0623 08:04:44.371259 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.371301 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5011900}
I20250623 08:04:44.370165 2210 leader_election.cc:290] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.371919 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16331ba6a34942a5a1b4ff4ac7bac55e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.375257 32647 raft_consensus.cc:2391] T 16331ba6a34942a5a1b4ff4ac7bac55e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:44.372370 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74da35cd34c14284af004001b25aa531" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.375638 1682 raft_consensus.cc:491] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.375780 1682 raft_consensus.cc:513] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.376374 1682 leader_election.cc:290] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.377211 339 ts_tablet_manager.cc:1428] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250623 08:04:44.377524 339 tablet_bootstrap.cc:492] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.377578 343 ts_tablet_manager.cc:1428] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.105s user 0.011s sys 0.006s
I20250623 08:04:44.378930 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb06a340f82047ac8ec3516edb08f4b6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.378875 32314 leader_election.cc:304] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:44.379127 32646 raft_consensus.cc:2391] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:44.379246 1682 raft_consensus.cc:3058] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.379474 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 351) took 5019 ms. Trace:
I20250623 08:04:44.379567 952 rpcz_store.cc:276] 0623 08:04:39.359687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.359776 (+ 89us) service_pool.cc:225] Handling call
0623 08:04:44.379366 (+5019590us) ts_tablet_manager.cc:1145] Deleting tablet 3249605158634f4eab5b9d61357c383f
0623 08:04:44.379375 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.379441 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5019442}
I20250623 08:04:44.379887 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 352) took 5019 ms. Trace:
I20250623 08:04:44.379963 952 rpcz_store.cc:276] 0623 08:04:39.360227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.360302 (+ 75us) service_pool.cc:225] Handling call
0623 08:04:44.379795 (+5019493us) ts_tablet_manager.cc:1145] Deleting tablet e9f8aa424bbb4aa8a907259270d46656
0623 08:04:44.379804 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.379858 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5019336}
I20250623 08:04:44.382521 339 tablet_bootstrap.cc:654] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.378895 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb06a340f82047ac8ec3516edb08f4b6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.377203 342 tablet_bootstrap.cc:492] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.383616 342 ts_tablet_manager.cc:1397] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.000s
I20250623 08:04:44.384388 342 raft_consensus.cc:357] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.384625 342 raft_consensus.cc:383] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.384742 342 raft_consensus.cc:738] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.385040 342 consensus_queue.cc:260] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.385958 343 tablet_bootstrap.cc:492] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.387471 343 tablet_bootstrap.cc:654] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.389963 1682 raft_consensus.cc:2747] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.391340 342 ts_tablet_manager.cc:1428] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250623 08:04:44.391726 342 tablet_bootstrap.cc:492] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:44.393482 32314 leader_election.cc:304] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:44.394991 1682 raft_consensus.cc:3058] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.395330 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 353) took 5034 ms. Trace:
I20250623 08:04:44.397501 1682 raft_consensus.cc:2747] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.397667 952 rpcz_store.cc:276] 0623 08:04:39.360764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.360820 (+ 56us) service_pool.cc:225] Handling call
0623 08:04:44.395200 (+5034380us) ts_tablet_manager.cc:1145] Deleting tablet c7fcf583064f4538913d9157e93068dd
0623 08:04:44.395216 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.395284 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5034240}
I20250623 08:04:44.398221 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 354) took 5036 ms. Trace:
I20250623 08:04:44.398511 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1f98d716ca3541379adbced08214bfdd [id=d1fbcd8e89164d71b021f4b1f4223a59]
I20250623 08:04:44.397969 342 tablet_bootstrap.cc:654] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:44.399168 32453 consensus_peers.cc:487] T affa3da7540648d28c5f00a3cf348eb1 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:44.400331 952 rpcz_store.cc:276] 0623 08:04:39.361345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.361436 (+ 91us) service_pool.cc:225] Handling call
0623 08:04:44.398108 (+5036672us) ts_tablet_manager.cc:1145] Deleting tablet 70f9d21b8c144f9cada9bdbe0fbb1538
0623 08:04:44.398118 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.398184 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5036534}
I20250623 08:04:44.401070 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_640be163e2114433bd7079aa3a602116 [id=d422f6202d1842e998a23ec064757334]
I20250623 08:04:44.401394 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 355) took 5039 ms. Trace:
I20250623 08:04:44.401566 952 rpcz_store.cc:276] 0623 08:04:39.361829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.361883 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:44.401275 (+5039392us) ts_tablet_manager.cc:1145] Deleting tablet b80a91ae2c084020b5d3444b570b28f2
0623 08:04:44.401283 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.401343 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5039250}
I20250623 08:04:44.402215 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]
I20250623 08:04:44.402655 952 tablet_replica.cc:331] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:44.402953 952 raft_consensus.cc:2241] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.407194 32314 leader_election.cc:304] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:44.408854 1682 raft_consensus.cc:3058] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.405326 1359 raft_consensus.cc:491] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.407030 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90ceb0f8d5c94c4e9427042d8a35e16c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:44.409641 32516 raft_consensus.cc:3058] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.408093 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bfa787f828b411ab68d81bb896f7fb6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:44.409847 32517 raft_consensus.cc:3058] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.412273 32516 raft_consensus.cc:2466] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:44.412520 1359 raft_consensus.cc:513] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.413152 32517 raft_consensus.cc:2466] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:44.413200 1359 leader_election.cc:290] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.415824 1682 raft_consensus.cc:2747] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.415920 952 raft_consensus.cc:2270] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.416564 952 ts_tablet_manager.cc:1905] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.417726 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11bb55088ab24b94af3e509810890247" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:44.418041 32517 raft_consensus.cc:3058] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.420629 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_23e6e7311fd242248a3adec872f67c93 [id=d71308abdeed4e7b9e623282d10aa15d]
I20250623 08:04:44.420984 32517 raft_consensus.cc:2466] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:44.421542 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1e293608b61743279ff20716c2953e24 [id=d7d9ed45091446fea8933157fd6e0db7]
I20250623 08:04:44.422503 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_fea0e0f4beec4bddb07a0d24efe95aa4 [id=d869eced4268436fbf4ba0666f3b7790]
I20250623 08:04:44.427563 343 tablet_bootstrap.cc:492] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.427774 343 ts_tablet_manager.cc:1397] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.004s
I20250623 08:04:44.428645 32589 leader_election.cc:304] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.429147 2013 raft_consensus.cc:2802] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.429267 2013 raft_consensus.cc:695] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.429539 2013 consensus_queue.cc:237] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.430721 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de743139c4324faba19ff3e5cfac855d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.432015 32646 raft_consensus.cc:2466] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:44.433334 339 tablet_bootstrap.cc:492] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.433570 339 ts_tablet_manager.cc:1397] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.002s
I20250623 08:04:44.434605 339 raft_consensus.cc:357] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.434890 339 raft_consensus.cc:383] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.435012 339 raft_consensus.cc:738] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.435379 339 consensus_queue.cc:260] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.435995 339 ts_tablet_manager.cc:1428] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:44.436199 343 raft_consensus.cc:357] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.436448 339 tablet_bootstrap.cc:492] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.437070 343 raft_consensus.cc:383] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.437283 343 raft_consensus.cc:738] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.437697 343 consensus_queue.cc:260] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.438908 343 ts_tablet_manager.cc:1428] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250623 08:04:44.439891 343 tablet_bootstrap.cc:492] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.440388 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_b08756509dd14eeeb598ef00e68600e2 [id=da8609b51169434c84a71ad27ca786b3]
I20250623 08:04:44.441306 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_744bf9c83e004aafb60d2cb44530dff5 [id=dd185047529940ebbd12e27a9f6577a6]
I20250623 08:04:44.441701 343 tablet_bootstrap.cc:654] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.442261 342 tablet_bootstrap.cc:492] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.442554 342 ts_tablet_manager.cc:1397] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.002s
I20250623 08:04:44.443616 342 raft_consensus.cc:357] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.443912 342 raft_consensus.cc:383] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.444031 342 raft_consensus.cc:738] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.444264 342 consensus_queue.cc:260] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.444809 342 ts_tablet_manager.cc:1428] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:44.445127 342 tablet_bootstrap.cc:492] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:44.447609 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_6a8ea07f0b174864b81addf035ed1f34 [id=dfff0b62abf54558b0b04a3b5d8cd896]
I20250623 08:04:44.449685 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e]
I20250623 08:04:44.450721 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]
I20250623 08:04:44.451313 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_8a32f131cc9048a583663a64ed26a4af [id=e201b6e61b4641b9b4894576ac00fad4]
I20250623 08:04:44.452132 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_88d6a5158a8d4ae192cf71e8eaffff54 [id=e218dda26a86485a9983a0bc74c4c89f]
I20250623 08:04:44.452900 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_fddbef2bc1d046689a8ef37cdcddcfa4 [id=e24b34aca192447a934b8775b709d8ac]
I20250623 08:04:44.453001 1682 raft_consensus.cc:491] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.453137 1682 raft_consensus.cc:513] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.453707 1682 leader_election.cc:290] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.454694 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de743139c4324faba19ff3e5cfac855d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.454878 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]
I20250623 08:04:44.455665 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]
I20250623 08:04:44.456679 981 ts_tablet_manager.cc:1918] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:44.456804 981 log.cc:1199] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/1a070895f4984493a3b9f6fa779797a9
I20250623 08:04:44.457183 981 ts_tablet_manager.cc:1939] T 1a070895f4984493a3b9f6fa779797a9 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.457890 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 352) took 5426 ms. Trace:
I20250623 08:04:44.457967 981 rpcz_store.cc:276] 0623 08:04:39.031719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.033219 (+ 1500us) service_pool.cc:225] Handling call
0623 08:04:44.254932 (+5221713us) ts_tablet_manager.cc:1145] Deleting tablet 1a070895f4984493a3b9f6fa779797a9
0623 08:04:44.254941 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.360243 (+105302us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.431520 (+ 71277us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.457872 (+ 26352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5187823}
I20250623 08:04:44.458232 981 tablet_replica.cc:331] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:44.458384 981 raft_consensus.cc:2241] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.458513 981 raft_consensus.cc:2270] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:44.459075 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 352) could send its response
W20250623 08:04:44.461073 32314 leader_election.cc:343] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 3740dd120745493688eeb4c1ac311dcb
I20250623 08:04:44.465459 343 tablet_bootstrap.cc:492] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.465670 343 ts_tablet_manager.cc:1397] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250623 08:04:44.441293 339 tablet_bootstrap.cc:654] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.466395 343 raft_consensus.cc:357] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.466632 343 raft_consensus.cc:383] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.466739 343 raft_consensus.cc:738] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.467041 343 consensus_queue.cc:260] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.467573 343 ts_tablet_manager.cc:1428] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:44.468010 343 tablet_bootstrap.cc:492] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.468621 981 ts_tablet_manager.cc:1905] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.469748 343 tablet_bootstrap.cc:654] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.481194 343 tablet_bootstrap.cc:492] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.481431 343 ts_tablet_manager.cc:1397] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20250623 08:04:44.482169 343 raft_consensus.cc:357] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.482465 343 raft_consensus.cc:383] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.482596 343 raft_consensus.cc:738] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.483186 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3740dd120745493688eeb4c1ac311dcb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.482988 343 consensus_queue.cc:260] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.484364 343 ts_tablet_manager.cc:1428] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:44.484970 343 tablet_bootstrap.cc:492] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.485538 32517 raft_consensus.cc:2391] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
W20250623 08:04:44.485926 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 347) could send its response
I20250623 08:04:44.486192 32314 leader_election.cc:304] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:44.486681 1682 raft_consensus.cc:3058] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.487751 1359 raft_consensus.cc:491] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 08:04:44.488355 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 349) could send its response
W20250623 08:04:44.488555 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 346) could send its response
I20250623 08:04:44.489630 1682 raft_consensus.cc:2747] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.495106 339 tablet_bootstrap.cc:492] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.495306 339 ts_tablet_manager.cc:1397] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.001s
I20250623 08:04:44.496519 339 raft_consensus.cc:357] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.496755 339 raft_consensus.cc:383] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.496879 339 raft_consensus.cc:738] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.497236 339 consensus_queue.cc:260] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.498225 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]
I20250623 08:04:44.498453 342 tablet_bootstrap.cc:654] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.498948 1359 raft_consensus.cc:513] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.499569 1359 leader_election.cc:290] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.501586 952 ts_tablet_manager.cc:1918] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:44.501765 339 ts_tablet_manager.cc:1428] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:44.502875 339 tablet_bootstrap.cc:492] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.503378 343 tablet_bootstrap.cc:654] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:44.506779 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 350) could send its response
W20250623 08:04:44.509683 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 351) could send its response
I20250623 08:04:44.510484 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_25f75684255a4b99a8aff38ee935aac5 [id=e85cbbd0d9db4945930b3fca419e3f2c]
I20250623 08:04:44.511174 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]
I20250623 08:04:44.511672 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_14e9f0f5ff5d42d9bdecf7f91482d8b1 [id=eb2ada6a6890451cb5eb869c176eecec]
I20250623 08:04:44.514259 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]
I20250623 08:04:44.514811 952 log.cc:1199] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/6926c49b787941d98c647543bd3b9ed5
I20250623 08:04:44.515496 952 ts_tablet_manager.cc:1939] T 6926c49b787941d98c647543bd3b9ed5 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:44.515012 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 352) could send its response
I20250623 08:04:44.516407 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_334749aabade478fbbc2badb1e4960bf [id=ec79bae33a55429fb05f25780369b398]
W20250623 08:04:44.516602 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 353) could send its response
W20250623 08:04:44.516888 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 354) could send its response
W20250623 08:04:44.517095 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 355) could send its response
I20250623 08:04:44.517658 2013 consensus_queue.cc:1035] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.518575 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6857437f85ab40f6a24e74ce3b5efbdc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.518592 32589 leader_election.cc:304] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.519398 32589 leader_election.cc:304] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.519102 339 tablet_bootstrap.cc:654] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.522356 2210 raft_consensus.cc:2802] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.522503 2210 raft_consensus.cc:695] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.522779 2210 consensus_queue.cc:237] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.523715 2188 raft_consensus.cc:2802] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.523825 2188 raft_consensus.cc:695] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.524055 2188 consensus_queue.cc:237] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.529074 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6857437f85ab40f6a24e74ce3b5efbdc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.531706 981 ts_tablet_manager.cc:1918] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:44.533145 981 log.cc:1199] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/9049e3d9b39f420bb51221808409fc54
I20250623 08:04:44.535177 32589 leader_election.cc:304] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.535434 2013 raft_consensus.cc:2802] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.535553 2013 raft_consensus.cc:695] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.535763 2013 consensus_queue.cc:237] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.536512 343 tablet_bootstrap.cc:492] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.536633 343 ts_tablet_manager.cc:1397] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20250623 08:04:44.537284 1957 raft_consensus.cc:491] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.537274 343 raft_consensus.cc:357] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.537420 343 raft_consensus.cc:383] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.537472 343 raft_consensus.cc:738] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.537456 1957 raft_consensus.cc:513] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.537636 343 consensus_queue.cc:260] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.538012 1957 leader_election.cc:290] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.538087 343 ts_tablet_manager.cc:1428] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:44.538326 343 tablet_bootstrap.cc:492] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.539614 981 ts_tablet_manager.cc:1939] T 9049e3d9b39f420bb51221808409fc54 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.540167 1957 consensus_queue.cc:1035] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.540832 32453 leader_election.cc:304] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:44.541360 1957 raft_consensus.cc:2802] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.541488 1957 raft_consensus.cc:491] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.541576 1957 raft_consensus.cc:3058] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.544247 1957 raft_consensus.cc:513] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.544797 1957 leader_election.cc:290] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.544597 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 354) took 5505 ms. Trace:
W20250623 08:04:44.545378 32453 leader_election.cc:343] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.545070 981 rpcz_store.cc:276] 0623 08:04:39.038691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.038743 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:44.458194 (+5419451us) ts_tablet_manager.cc:1145] Deleting tablet 9049e3d9b39f420bb51221808409fc54
0623 08:04:44.458202 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.493560 (+ 35358us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.530870 (+ 37310us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.544422 (+ 13552us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5387535}
I20250623 08:04:44.545979 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 356) took 5183 ms. Trace:
W20250623 08:04:44.546240 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 354) could send its response
I20250623 08:04:44.546190 952 rpcz_store.cc:276] 0623 08:04:39.362336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.362436 (+ 100us) service_pool.cc:225] Handling call
0623 08:04:44.402588 (+5040152us) ts_tablet_manager.cc:1145] Deleting tablet 6926c49b787941d98c647543bd3b9ed5
0623 08:04:44.402597 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.438649 (+ 36052us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.501459 (+ 62810us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.545921 (+ 44462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5039985}
I20250623 08:04:44.546685 952 tablet_replica.cc:331] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:44.546972 952 raft_consensus.cc:2241] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:44.547035 343 tablet_bootstrap.cc:654] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.547174 981 tablet_replica.cc:331] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:44.547735 952 raft_consensus.cc:2270] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.548127 2014 consensus_queue.cc:1035] T 344ebe1a14bc4d3389b0161ff102f05c P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.548175 981 raft_consensus.cc:2241] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.551007 952 ts_tablet_manager.cc:1905] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.551883 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea588a48e6164d3db148fdaa306268f6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.552867 981 raft_consensus.cc:2270] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.553915 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea588a48e6164d3db148fdaa306268f6" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.559859 981 ts_tablet_manager.cc:1905] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.559518 1957 raft_consensus.cc:491] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:44.572856 1957 raft_consensus.cc:513] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.573546 1957 leader_election.cc:290] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.573593 1682 raft_consensus.cc:491] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.573800 1682 raft_consensus.cc:513] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.574383 1682 leader_election.cc:290] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.575879 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f217a8eb12014fcf89b9df4a2cddf5f6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.575897 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f217a8eb12014fcf89b9df4a2cddf5f6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.576632 32314 leader_election.cc:304] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:44.576956 1682 raft_consensus.cc:2747] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.577759 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0b1287cf3b6459eafd032afc77c5824" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.578594 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0b1287cf3b6459eafd032afc77c5824" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.578945 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c]
I20250623 08:04:44.579159 2210 raft_consensus.cc:491] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.579345 2210 raft_consensus.cc:513] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.579988 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_e9f2ab1d95fb4341b5d05f6f5bd09f15 [id=f0a8652a629545a9a9bc4c22aba0da74]
I20250623 08:04:44.579984 2210 leader_election.cc:290] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.580539 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59015204545e436cb558fe4465a67c97" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.581003 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_eb0f697365cc4f2b8bbeb5b3de47f1dc [id=f15f4f7ed5054a3ca292217b18a14a94]
W20250623 08:04:44.581174 32589 leader_election.cc:343] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.581506 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_849f3be400444579943f3f649824f4ea [id=f223e5f800b14f08b72352e85db1a9f7]
I20250623 08:04:44.602778 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de743139c4324faba19ff3e5cfac855d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:44.602979 32647 raft_consensus.cc:3058] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.603348 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b]
I20250623 08:04:44.603873 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe [id=f77e622d3eb44f4fa3a284b08eefc109]
I20250623 08:04:44.604692 2076 catalog_manager.cc:671] Loaded metadata for table default.Table_d3115e0a02304f6d80807841b76504e4 [id=fc36d4cdd9684c6f84517d848ae3395b]
I20250623 08:04:44.605764 32647 raft_consensus.cc:2466] T de743139c4324faba19ff3e5cfac855d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:44.606655 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 356) could send its response
I20250623 08:04:44.607292 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de743139c4324faba19ff3e5cfac855d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.608059 343 tablet_bootstrap.cc:492] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.609153 343 ts_tablet_manager.cc:1397] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.000s
I20250623 08:04:44.611543 2188 raft_consensus.cc:491] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.611732 2188 raft_consensus.cc:513] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.613123 32381 raft_consensus.cc:3058] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.613135 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34da529ffde4595a23fc6596b30e3e9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.613673 2188 leader_election.cc:290] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.619247 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59015204545e436cb558fe4465a67c97" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.619482 32516 raft_consensus.cc:2466] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.632916 342 tablet_bootstrap.cc:492] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.633145 342 ts_tablet_manager.cc:1397] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.188s user 0.005s sys 0.004s
I20250623 08:04:44.634454 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34da529ffde4595a23fc6596b30e3e9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:44.634543 32453 leader_election.cc:343] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.615028 343 raft_consensus.cc:357] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.634690 32517 raft_consensus.cc:2466] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
W20250623 08:04:44.617241 32589 leader_election.cc:343] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:44.635454 32453 leader_election.cc:343] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.635577 32453 leader_election.cc:304] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.618816 2013 raft_consensus.cc:491] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.635644 2210 consensus_queue.cc:1035] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.635807 343 raft_consensus.cc:383] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.635877 343 raft_consensus.cc:738] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.635761 2013 raft_consensus.cc:513] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.636039 343 consensus_queue.cc:260] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.636353 2013 leader_election.cc:290] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.636602 981 ts_tablet_manager.cc:1918] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:44.635753 2242 raft_consensus.cc:491] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.636721 981 log.cc:1199] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/4c9d1c8a67e149478a0d69842a4eb5b2
I20250623 08:04:44.636767 2242 raft_consensus.cc:513] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.637084 981 ts_tablet_manager.cc:1939] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.637285 2242 leader_election.cc:290] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.637729 343 ts_tablet_manager.cc:1428] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250623 08:04:44.637972 343 tablet_bootstrap.cc:492] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.638882 1682 raft_consensus.cc:491] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.639029 1682 raft_consensus.cc:513] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.639496 343 tablet_bootstrap.cc:654] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.639573 1682 leader_election.cc:290] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.640545 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 355) took 5599 ms. Trace:
I20250623 08:04:44.640851 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f957f5575e9475eb9f639e3ad5cb809" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.640849 981 rpcz_store.cc:276] 0623 08:04:39.040723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.040769 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:44.546963 (+5506194us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d1c8a67e149478a0d69842a4eb5b2
0623 08:04:44.546976 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.588858 (+ 41882us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.636483 (+ 47625us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.640435 (+ 3952us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5475716}
I20250623 08:04:44.641438 342 raft_consensus.cc:357] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.641602 342 raft_consensus.cc:383] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.641702 342 raft_consensus.cc:738] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.641532 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0107b41d5c04a5c818f6792f2c18ce5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.641919 342 consensus_queue.cc:260] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.642804 981 tablet_replica.cc:331] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:44.643507 339 tablet_bootstrap.cc:492] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.643831 339 ts_tablet_manager.cc:1397] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.142s user 0.003s sys 0.015s
I20250623 08:04:44.644483 339 raft_consensus.cc:357] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.644707 339 raft_consensus.cc:383] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.644820 339 raft_consensus.cc:738] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.645159 339 consensus_queue.cc:260] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.645732 339 ts_tablet_manager.cc:1428] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:44.646059 339 tablet_bootstrap.cc:492] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:44.643859 32589 leader_election.cc:343] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:44.647948 32589 leader_election.cc:343] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.648739 981 raft_consensus.cc:2241] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:44.649375 32589 consensus_peers.cc:487] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9049e3d9b39f420bb51221808409fc54. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:44.650063 981 raft_consensus.cc:2270] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.655759 339 tablet_bootstrap.cc:654] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.655921 981 ts_tablet_manager.cc:1905] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.658633 342 ts_tablet_manager.cc:1428] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250623 08:04:44.659070 1953 raft_consensus.cc:2747] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.660725 342 tablet_bootstrap.cc:492] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:44.662653 342 tablet_bootstrap.cc:654] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:44.660148 32453 leader_election.cc:343] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:44.666204 32453 consensus_peers.cc:487] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:44.666626 32453 leader_election.cc:343] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:44.666903 32453 leader_election.cc:343] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.675721 32589 leader_election.cc:304] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.675977 2013 raft_consensus.cc:2802] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.676043 2013 raft_consensus.cc:491] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.676092 2013 raft_consensus.cc:3058] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.678460 2013 raft_consensus.cc:513] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.679067 2013 leader_election.cc:290] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.679876 32589 leader_election.cc:304] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.686997 2242 raft_consensus.cc:2802] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.687140 2242 raft_consensus.cc:491] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.687255 2242 raft_consensus.cc:3058] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.689929 2242 raft_consensus.cc:513] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.690538 2242 leader_election.cc:290] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.665086 952 ts_tablet_manager.cc:1918] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:44.691766 952 log.cc:1199] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/f02f36d6c1174a23a1f1043a431d0b6b
I20250623 08:04:44.692904 952 ts_tablet_manager.cc:1939] T f02f36d6c1174a23a1f1043a431d0b6b P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:44.694834 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 337) took 5348 ms. Trace:
I20250623 08:04:44.695566 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59015204545e436cb558fe4465a67c97" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.695866 339 tablet_bootstrap.cc:492] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.695993 339 ts_tablet_manager.cc:1397] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.004s
W20250623 08:04:44.697319 32589 leader_election.cc:343] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.696882 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34da529ffde4595a23fc6596b30e3e9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.699283 343 tablet_bootstrap.cc:492] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.699482 343 ts_tablet_manager.cc:1397] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.004s
I20250623 08:04:44.700233 343 raft_consensus.cc:357] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.700474 343 raft_consensus.cc:383] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.700584 343 raft_consensus.cc:738] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.700865 343 consensus_queue.cc:260] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.700897 952 rpcz_store.cc:276] 0623 08:04:39.346293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.372266 (+ 25973us) service_pool.cc:225] Handling call
0623 08:04:44.546594 (+5174328us) ts_tablet_manager.cc:1145] Deleting tablet f02f36d6c1174a23a1f1043a431d0b6b
0623 08:04:44.546607 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.652014 (+105407us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.664945 (+ 12931us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.694729 (+ 29784us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5173965}
W20250623 08:04:44.701437 32589 leader_election.cc:343] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:44.692113 32453 leader_election.cc:343] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.701591 32453 leader_election.cc:304] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.701886 1359 raft_consensus.cc:2747] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.701932 343 ts_tablet_manager.cc:1428] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:44.702344 343 tablet_bootstrap.cc:492] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.703023 1359 consensus_queue.cc:1035] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:44.703617 32453 leader_election.cc:343] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.703739 32453 leader_election.cc:304] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.703976 1359 raft_consensus.cc:2747] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.704319 32453 leader_election.cc:304] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.706202 342 tablet_bootstrap.cc:492] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.706372 342 ts_tablet_manager.cc:1397] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.004s
I20250623 08:04:44.706967 1359 raft_consensus.cc:2802] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.707119 1359 raft_consensus.cc:695] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:44.707386 1359 consensus_queue.cc:237] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.715931 981 ts_tablet_manager.cc:1918] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:44.718986 343 tablet_bootstrap.cc:654] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:44.720530 32453 leader_election.cc:343] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: aa398354a9e44633857a75c0bbba08b3
I20250623 08:04:44.721117 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 335) took 5369 ms. Trace:
I20250623 08:04:44.698668 339 raft_consensus.cc:357] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.721419 952 rpcz_store.cc:276] 0623 08:04:39.331572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.372736 (+ 41164us) service_pool.cc:225] Handling call
0623 08:04:44.701209 (+5328473us) ts_tablet_manager.cc:1145] Deleting tablet f0e24d8d890d4d1db051f172b3ec96ec
0623 08:04:44.701220 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.701333 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5328129}
I20250623 08:04:44.721660 981 log.cc:1199] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/a8945b6f49474648ac73bcb85285680a
I20250623 08:04:44.722776 1957 raft_consensus.cc:491] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.723001 1957 raft_consensus.cc:513] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.723585 1957 leader_election.cc:290] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.723986 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 357) took 5361 ms. Trace:
I20250623 08:04:44.724082 981 ts_tablet_manager.cc:1939] T a8945b6f49474648ac73bcb85285680a P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.722051 339 raft_consensus.cc:383] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.725651 952 rpcz_store.cc:276] 0623 08:04:39.405113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.405304 (+ 191us) service_pool.cc:225] Handling call
0623 08:04:44.723865 (+5318561us) ts_tablet_manager.cc:1145] Deleting tablet ecb422550efd40cf82bc92fca5c25eda
0623 08:04:44.723875 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.723956 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5318349}
I20250623 08:04:44.725293 339 raft_consensus.cc:738] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.726109 1957 consensus_queue.cc:1035] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.727066 339 consensus_queue.cc:260] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.730530 2076 tablet_loader.cc:96] loaded metadata for tablet 035bb06bc7dd42f681886def9d56eb73 (table default.Table_905d0330c27149a19dabb586c51a2e0a [id=b3f407a7a26e4bd5ae9a719a5c4577c0])
I20250623 08:04:44.730978 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 358) took 5324 ms. Trace:
I20250623 08:04:44.731097 952 rpcz_store.cc:276] 0623 08:04:39.406590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.406873 (+ 283us) service_pool.cc:225] Handling call
0623 08:04:44.730860 (+5323987us) ts_tablet_manager.cc:1145] Deleting tablet 52d0a356c74f4daea9093c998b4bf73e
0623 08:04:44.730870 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.730938 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5323769}
I20250623 08:04:44.731837 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f957f5575e9475eb9f639e3ad5cb809" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.733296 32517 raft_consensus.cc:2466] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.731952 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0107b41d5c04a5c818f6792f2c18ce5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.733543 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59015204545e436cb558fe4465a67c97" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:44.733726 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 359) took 5325 ms. Trace:
I20250623 08:04:44.733777 32513 raft_consensus.cc:3058] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.733834 952 rpcz_store.cc:276] 0623 08:04:39.409448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.409794 (+ 346us) service_pool.cc:225] Handling call
0623 08:04:44.733358 (+5323564us) ts_tablet_manager.cc:1145] Deleting tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f
0623 08:04:44.733367 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.733687 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5323326}
I20250623 08:04:44.726608 342 raft_consensus.cc:357] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.733584 32514 raft_consensus.cc:2466] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.734195 342 raft_consensus.cc:383] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.734284 342 raft_consensus.cc:738] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.734586 342 consensus_queue.cc:260] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.732731 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34da529ffde4595a23fc6596b30e3e9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:44.735083 32518 raft_consensus.cc:3058] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.734611 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 358) took 5604 ms. Trace:
I20250623 08:04:44.736035 342 ts_tablet_manager.cc:1428] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20250623 08:04:44.736358 342 tablet_bootstrap.cc:492] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:44.735291 981 rpcz_store.cc:276] 0623 08:04:39.130551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.131515 (+ 964us) service_pool.cc:225] Handling call
0623 08:04:44.642651 (+5511136us) ts_tablet_manager.cc:1145] Deleting tablet a8945b6f49474648ac73bcb85285680a
0623 08:04:44.642662 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.692328 (+ 49666us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.715818 (+ 23490us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.734589 (+ 18771us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5510774}
I20250623 08:04:44.736788 339 ts_tablet_manager.cc:1428] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20250623 08:04:44.737884 32518 raft_consensus.cc:2466] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:44.738117 342 tablet_bootstrap.cc:654] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.738907 981 tablet_replica.cc:331] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:44.739506 339 tablet_bootstrap.cc:492] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.740844 981 raft_consensus.cc:2241] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.742733 981 raft_consensus.cc:2270] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.751331 343 tablet_bootstrap.cc:492] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.751586 343 ts_tablet_manager.cc:1397] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.001s
I20250623 08:04:44.751524 981 ts_tablet_manager.cc:1905] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.752542 343 raft_consensus.cc:357] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.752787 343 raft_consensus.cc:383] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.752887 343 raft_consensus.cc:738] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.753165 343 consensus_queue.cc:260] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.753806 343 ts_tablet_manager.cc:1428] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:44.754244 343 tablet_bootstrap.cc:492] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.756888 343 tablet_bootstrap.cc:654] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.757196 339 tablet_bootstrap.cc:654] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.760273 2076 tablet_loader.cc:96] loaded metadata for tablet 049b9194c9f74b8997c06469d74b9795 (table default.Table_4d2fc00363204f8485905af018f7911a [id=22f134d5d55f40ef98dec2f14a6d55ce])
I20250623 08:04:44.761159 2076 tablet_loader.cc:96] loaded metadata for tablet 04fde7d1c1744c0597e43086ca4c9541 (table default.Table_06b731510ae743608f1fbc4d250acf14 [id=7fbe15151173464ea0ee480720c58b57])
I20250623 08:04:44.761520 2076 tablet_loader.cc:96] loaded metadata for tablet 0545daa6fba846adbc573a63d8dbcd66 (table default.Table_e69f9fec5b604bca8658b2124145f92e [id=96cf904f9ea64544ae6f662208874f89])
I20250623 08:04:44.761862 2076 tablet_loader.cc:96] loaded metadata for tablet 0571fa96923144d5ac39429d252c8723 (table default.Table_1ef88ab1f1cd49139106aed3732b50a8 [id=90865844914743379da60c53e1c3a0d5])
I20250623 08:04:44.762235 2076 tablet_loader.cc:96] loaded metadata for tablet 0749cb735acf4cb393ad78a2cd0c0d60 (table default.Table_7a30e48550d443ccbe36157dd0e35bcc [id=7a1f259c8fc24b4ca83cfa09e5673d58])
W20250623 08:04:44.738308 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 337) could send its response
I20250623 08:04:44.740533 32513 raft_consensus.cc:2466] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
W20250623 08:04:44.764547 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 335) could send its response
I20250623 08:04:44.773545 2076 tablet_loader.cc:96] loaded metadata for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b])
I20250623 08:04:44.774130 2076 tablet_loader.cc:96] loaded metadata for tablet 07aac1a69513468f80f40d295853279a (table default.Table_b08756509dd14eeeb598ef00e68600e2 [id=da8609b51169434c84a71ad27ca786b3])
I20250623 08:04:44.774564 2076 tablet_loader.cc:96] loaded metadata for tablet 081cef8b055341dbb98d6de3c918d2e2 (table default.Table_1911a70d3e224ae4b9805e345c99624b [id=9f504d8065a24602af6e2142069438a5])
I20250623 08:04:44.775048 2076 tablet_loader.cc:96] loaded metadata for tablet 0861107ab4624a05bd7a4d24b420c502 (table default.Table_977fe8fc665f4267b394065e22f67eea [id=be9754ecb40a413d9940224adbd7e2eb])
I20250623 08:04:44.775501 2076 tablet_loader.cc:96] loaded metadata for tablet 0861e45cf546487d9812caa06e2704eb (table default.Table_1671bbefe7bd4d4b86a917373288a1a6 [id=baaeae53fdcb49b7889d8379fa203c79])
I20250623 08:04:44.775970 2076 tablet_loader.cc:96] loaded metadata for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e])
I20250623 08:04:44.776456 2076 tablet_loader.cc:96] loaded metadata for tablet 0c58231f67604422a1497e4c1b44ec52 (table default.Table_fe81102858f244af9a4a286eaf6e34bf [id=8a0d9f5619d843148169561ffc7cfdc0])
I20250623 08:04:44.777112 2076 tablet_loader.cc:96] loaded metadata for tablet 0cca63991d5b4ef5bd8a9181d127ebe3 (table default.Table_da00c692e65345c496f4e52c106aa2e6 [id=5ea293756d514f06b8d749c615417969])
I20250623 08:04:44.777647 2076 tablet_loader.cc:96] loaded metadata for tablet 0db9480bb07343fe935fe9bfc98bca77 (table default.Table_1f98d716ca3541379adbced08214bfdd [id=d1fbcd8e89164d71b021f4b1f4223a59])
I20250623 08:04:44.778172 2076 tablet_loader.cc:96] loaded metadata for tablet 0e9bf7cba2244703b7402cb4be1a005e (table default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c])
I20250623 08:04:44.778545 2076 tablet_loader.cc:96] loaded metadata for tablet 0eec43a08c9c4211aa2cf53ff54b88f5 (table default.Table_14e9f0f5ff5d42d9bdecf7f91482d8b1 [id=eb2ada6a6890451cb5eb869c176eecec])
I20250623 08:04:44.779667 2076 tablet_loader.cc:96] loaded metadata for tablet 115763354f294954a260b0aa5c66a146 (table default.Table_973ac8c9b50b4fb2a8d83318384a8b76 [id=ab80a8647b644e6ebbdfba0d2e76107a])
I20250623 08:04:44.780043 2076 tablet_loader.cc:96] loaded metadata for tablet 11bb55088ab24b94af3e509810890247 (table default.Table_49e1ee7e0d024625988be24a823a443c [id=9a7a76e61e1b43508b6f6080f5163417])
I20250623 08:04:44.781020 2076 tablet_loader.cc:96] loaded metadata for tablet 13824916241c4cf596119d44d89cd17f (table default.Table_c491a58f8bb24bbf85189ea654bdd45f [id=826cd9624f2a4f818f02427a9462a9b2])
I20250623 08:04:44.781389 2076 tablet_loader.cc:96] loaded metadata for tablet 1494f83d13cd49c7a804593f45f97cc6 (table default.Table_d87c4052e62d4c1c87fc25cff314319b [id=58d5dec0c723470a9a7e56ce37dc71d0])
I20250623 08:04:44.781790 2076 tablet_loader.cc:96] loaded metadata for tablet 15154c85de7144a9b9e2367c70f57e2e (table default.Table_7c712a75a62f41dba90fcf504f4f752c [id=780a3bc4b3004e06a840a0b63e0c37c6])
I20250623 08:04:44.782163 2076 tablet_loader.cc:96] loaded metadata for tablet 15d9b4e331414d3eace2b4098b13731e (table default.Table_070eb9d8f3af4ed9bb823cf579ade9c8 [id=8f90ed2f56464b9f8dc494452e6c6aed])
I20250623 08:04:44.783761 339 tablet_bootstrap.cc:492] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.783972 339 ts_tablet_manager.cc:1397] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.001s
I20250623 08:04:44.784780 339 raft_consensus.cc:357] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.785017 339 raft_consensus.cc:383] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.785115 339 raft_consensus.cc:738] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.785377 339 consensus_queue.cc:260] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.785519 1359 raft_consensus.cc:491] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.785662 1359 raft_consensus.cc:513] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.786260 1359 leader_election.cc:290] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:44.787451 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 357) could send its response
I20250623 08:04:44.787803 2076 tablet_loader.cc:96] loaded metadata for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47])
I20250623 08:04:44.788959 339 ts_tablet_manager.cc:1428] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250623 08:04:44.789465 339 tablet_bootstrap.cc:492] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.792461 339 tablet_bootstrap.cc:654] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.793663 2076 tablet_loader.cc:96] loaded metadata for tablet 1793340e45b74e4aba3f57c20cc54903 (table default.Table_8ab4ba6029964c2fa123202d6c5c836d [id=3c1572742d694d48be03ccabe9166f8d])
I20250623 08:04:44.797715 2076 tablet_loader.cc:96] loaded metadata for tablet 1a404aac062c41e8a06207d9d8685a4d (table default.Table_ac62bae678de4b11a971f7bbccd50303 [id=4f4663a5cad14a3593e16a52ec8aeece])
W20250623 08:04:44.797971 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 358) could send its response
I20250623 08:04:44.799381 2160 raft_consensus.cc:491] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.799510 2160 raft_consensus.cc:513] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.800092 2160 leader_election.cc:290] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.803777 2160 consensus_queue.cc:1035] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.805153 1957 consensus_queue.cc:1035] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.805755 1957 consensus_queue.cc:1035] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.808348 2076 tablet_loader.cc:96] loaded metadata for tablet 1ae0a411a3e54c45b8d5c5b9b0eea762 (table default.Table_71f2f0414e0840be832096955576cb2f [id=a6918c5eb39a435ab4b035c1509916a0])
I20250623 08:04:44.812125 2076 tablet_loader.cc:96] loaded metadata for tablet 1c529e826fa84a919c018887620e4023 (table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166])
I20250623 08:04:44.816964 981 ts_tablet_manager.cc:1918] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:44.817881 981 log.cc:1199] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/f02f36d6c1174a23a1f1043a431d0b6b
I20250623 08:04:44.819406 2076 tablet_loader.cc:96] loaded metadata for tablet 1cee3de5953244f98428747bea912f77 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc])
I20250623 08:04:44.821810 2076 tablet_loader.cc:96] loaded metadata for tablet 1ed334ed653d4dea827f4e6ce9e1bd8b (table default.Table_f0683327c2bf468582a2f3e557d15725 [id=acb1a73cc8b546dab7431a57c3a8857e])
I20250623 08:04:44.823721 2076 tablet_loader.cc:96] loaded metadata for tablet 1f76ef24c69f4b17a373b242d71da8b9 (table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d])
I20250623 08:04:44.824638 2076 tablet_loader.cc:96] loaded metadata for tablet 207c75acfed64082a885b47ae5353c9d (table default.Table_c67b6294376343b38de7a2aa4ed7bcea [id=898fc58a25554c09867e70728f4a7508])
I20250623 08:04:44.765578 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cd316e6f4447be958611ab07ed31fc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.803572 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c74e8c697d244ac2ac8174bb1fc76e61" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.806126 2014 raft_consensus.cc:491] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.809238 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55669db7e55042b2bc208ea3218c2d6c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.825786 32654 raft_consensus.cc:2466] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:44.825866 2076 tablet_loader.cc:96] loaded metadata for tablet 21080ab069d74457a9f7fe3aabdb3c7c (table default.Table_f1d8f7ef7cf144eaa1fd741b10eac3d6 [id=9060c1640fca461c9020722b3165d144])
I20250623 08:04:44.826155 2014 raft_consensus.cc:513] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.825824 981 ts_tablet_manager.cc:1939] T f02f36d6c1174a23a1f1043a431d0b6b P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.826508 32645 raft_consensus.cc:2466] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:44.827507 2014 leader_election.cc:290] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.818892 32589 leader_election.cc:304] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.827980 32589 leader_election.cc:304] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.828472 32589 leader_election.cc:304] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.828739 2013 raft_consensus.cc:2802] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.828832 2013 raft_consensus.cc:491] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.828924 2013 raft_consensus.cc:3058] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.829335 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b11777e0bd04e61a5bc8bb6ec4d97c3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:44.831331 339 tablet_bootstrap.cc:492] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.831444 2013 raft_consensus.cc:513] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.831970 2076 tablet_loader.cc:96] loaded metadata for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351])
I20250623 08:04:44.832157 339 ts_tablet_manager.cc:1397] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20250623 08:04:44.832448 2013 leader_election.cc:290] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.832624 2013 raft_consensus.cc:2802] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.832844 2013 raft_consensus.cc:695] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.832949 339 raft_consensus.cc:357] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.833107 2013 consensus_queue.cc:237] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.833491 2076 tablet_loader.cc:96] loaded metadata for tablet 2414d24ab6c54f36bb1e554f7231a5d7 (table default.Table_a1f91329476d4e4f9be66462bda04dbf [id=f69e9adaf68a47f2a80d305c1936333b])
I20250623 08:04:44.833966 339 raft_consensus.cc:383] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.834074 339 raft_consensus.cc:738] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.828151 2014 raft_consensus.cc:2802] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.834175 2014 raft_consensus.cc:491] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.834237 2076 tablet_loader.cc:96] loaded metadata for tablet 257227ae140446d78117b6150dbb3962 (table default.Table_bd4ee5f643fe4d9bab6b2f11ef0869cf [id=28106e3ea0bd442e85bbbe3e26f8f0d4])
I20250623 08:04:44.834340 2014 raft_consensus.cc:3058] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.834465 339 consensus_queue.cc:260] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.835134 342 tablet_bootstrap.cc:492] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.835952 2076 tablet_loader.cc:96] loaded metadata for tablet 28d79c96bd1c4ddfb4a1dd8a9d83c3fe (table default.Table_23e6e7311fd242248a3adec872f67c93 [id=d71308abdeed4e7b9e623282d10aa15d])
I20250623 08:04:44.834805 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 359) took 5703 ms. Trace:
I20250623 08:04:44.836161 981 rpcz_store.cc:276] 0623 08:04:39.131610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.131760 (+ 150us) service_pool.cc:225] Handling call
0623 08:04:44.738606 (+5606846us) ts_tablet_manager.cc:1145] Deleting tablet f02f36d6c1174a23a1f1043a431d0b6b
0623 08:04:44.738618 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.770805 (+ 32187us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.816366 (+ 45561us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.834784 (+ 18418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5606416}
I20250623 08:04:44.836740 342 ts_tablet_manager.cc:1397] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.000s
I20250623 08:04:44.836978 981 tablet_replica.cc:331] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:44.837064 2014 raft_consensus.cc:513] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.837155 981 raft_consensus.cc:2241] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:44.837409 981 raft_consensus.cc:2270] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.837698 2014 leader_election.cc:290] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.837972 981 ts_tablet_manager.cc:1905] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.838034 342 raft_consensus.cc:357] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.838130 32589 leader_election.cc:304] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:44.838619 2242 raft_consensus.cc:2802] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.836771 339 ts_tablet_manager.cc:1428] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:44.838719 2242 raft_consensus.cc:695] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.839001 339 tablet_bootstrap.cc:492] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.838997 2242 consensus_queue.cc:237] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.840658 339 tablet_bootstrap.cc:654] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.842182 2076 tablet_loader.cc:96] loaded metadata for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90])
I20250623 08:04:44.842711 2076 tablet_loader.cc:96] loaded metadata for tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda])
I20250623 08:04:44.843499 2076 tablet_loader.cc:96] loaded metadata for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c])
I20250623 08:04:44.844270 2076 tablet_loader.cc:96] loaded metadata for tablet 2ec871b2878a4b3f929b3774746d7750 (table default.Table_787415ae5937475f90f90000aa575d0e [id=4b27489168854629848531915104847a])
I20250623 08:04:44.845121 2076 tablet_loader.cc:96] loaded metadata for tablet 2f7ba3b78aab4871a8f07e4ecc24acd4 (table default.Table_100f074733ca4066983e68257aea0b11 [id=5d78d651cb7345fba13c371f9c1db55a])
I20250623 08:04:44.845371 342 raft_consensus.cc:383] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.845494 2076 tablet_loader.cc:96] loaded metadata for tablet 3054f406c1c34f5d92edeae1b9243777 (table default.Table_a935cce17e794f328640dde460d22086 [id=5847481518254b318e71e8198962267a])
I20250623 08:04:44.845494 342 raft_consensus.cc:738] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.845741 342 consensus_queue.cc:260] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.845901 2076 tablet_loader.cc:96] loaded metadata for tablet 306c4021b5d94a4996de07c222a8bfd8 (table default.Table_cdabecebee2e4010a7325455f785d002 [id=a52743896ed04011a1662f3608454ff4])
I20250623 08:04:44.846510 342 ts_tablet_manager.cc:1428] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250623 08:04:44.847225 342 tablet_bootstrap.cc:492] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:44.848753 342 tablet_bootstrap.cc:654] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.852273 981 ts_tablet_manager.cc:1918] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:44.852432 981 log.cc:1199] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/f4a96ae9190640dea11bd11a9f0e4372
I20250623 08:04:44.852908 981 ts_tablet_manager.cc:1939] T f4a96ae9190640dea11bd11a9f0e4372 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.859850 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b11777e0bd04e61a5bc8bb6ec4d97c3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:44.859849 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c74e8c697d244ac2ac8174bb1fc76e61" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.860116 32645 raft_consensus.cc:2391] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:44.860884 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0107b41d5c04a5c818f6792f2c18ce5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.861642 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f957f5575e9475eb9f639e3ad5cb809" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.862246 2188 raft_consensus.cc:491] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.862363 2188 raft_consensus.cc:513] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.863005 2188 leader_election.cc:290] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.864542 32381 raft_consensus.cc:3058] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.864717 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 360) took 5732 ms. Trace:
W20250623 08:04:44.866077 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 355) could send its response
I20250623 08:04:44.865247 981 rpcz_store.cc:276] 0623 08:04:39.131719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.131920 (+ 201us) service_pool.cc:225] Handling call
0623 08:04:44.836925 (+5705005us) ts_tablet_manager.cc:1145] Deleting tablet f4a96ae9190640dea11bd11a9f0e4372
0623 08:04:44.836936 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.849219 (+ 12283us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.852173 (+ 2954us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.864404 (+ 12231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5704874}
I20250623 08:04:44.868185 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 357) took 5738 ms. Trace:
I20250623 08:04:44.868846 981 rpcz_store.cc:276] 0623 08:04:39.129866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.132238 (+ 2372us) service_pool.cc:225] Handling call
0623 08:04:44.867460 (+5735222us) ts_tablet_manager.cc:1145] Deleting tablet f0e24d8d890d4d1db051f172b3ec96ec
0623 08:04:44.867474 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.868013 (+ 539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5734823}
I20250623 08:04:44.869685 2076 tablet_loader.cc:96] loaded metadata for tablet 31a67b70cbc64bb3bfd5a34865109a01 (table default.Table_64fe0b4300ff490ea1b7ebf1333ecc5e [id=7a569a9fe3bd4c608ddca48a734df1da])
I20250623 08:04:44.870412 2076 tablet_loader.cc:96] loaded metadata for tablet 31ecc74317d449e5b557b07a4fd14895 (table default.Table_71fd2906f6b0404580099cb500e1f7fb [id=6e61ff2d881e41dcbf03374e0f48b4ca])
I20250623 08:04:44.870704 981 tablet_replica.cc:331] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: stopping tablet replica
W20250623 08:04:44.871349 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 358) could send its response
I20250623 08:04:44.872400 981 raft_consensus.cc:2241] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:44.873088 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa398354a9e44633857a75c0bbba08b3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.873268 32378 raft_consensus.cc:2466] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:44.874454 339 tablet_bootstrap.cc:492] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.874625 339 ts_tablet_manager.cc:1397] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20250623 08:04:44.874867 2076 tablet_loader.cc:96] loaded metadata for tablet 344ebe1a14bc4d3389b0161ff102f05c (table default.Table_da6195995899479b879a3d41e0c07ed4 [id=b3b1a699a6b44ff09cf7b4f105e04794])
I20250623 08:04:44.874682 981 raft_consensus.cc:2270] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:44.875353 339 raft_consensus.cc:357] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.875908 339 raft_consensus.cc:383] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.876544 32378 raft_consensus.cc:3058] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.877158 339 raft_consensus.cc:738] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.877408 339 consensus_queue.cc:260] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.879261 981 ts_tablet_manager.cc:1905] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:44.876679 32381 raft_consensus.cc:3058] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:44.889705 32589 leader_election.cc:343] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:44.889988 32589 leader_election.cc:343] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:44.890211 32589 leader_election.cc:343] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.891105 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3136ec837bea4619ae3995ceec9b24d6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.891371 32381 raft_consensus.cc:2466] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.894203 343 tablet_bootstrap.cc:492] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:44.896148 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c74e8c697d244ac2ac8174bb1fc76e61" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.896653 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55669db7e55042b2bc208ea3218c2d6c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.896595 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cd316e6f4447be958611ab07ed31fc" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:44.897714 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 359) could send its response
I20250623 08:04:44.897951 2076 tablet_loader.cc:96] loaded metadata for tablet 36d21fe528f5410eacd655bf15306b98 (table default.Table_a50378b31cda420a8791443844eed5f2 [id=b83148d602ab4c19a5fb1b4072b626e0])
W20250623 08:04:44.898773 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 360) could send its response
W20250623 08:04:44.898908 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 357) could send its response
I20250623 08:04:44.900516 339 ts_tablet_manager.cc:1428] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250623 08:04:44.900851 339 tablet_bootstrap.cc:492] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.902472 2076 tablet_loader.cc:96] loaded metadata for tablet 372a29156dab44f59d8ff435a51bdfcd (table default.Table_4903e809cc2d447fb94224ccbb05078d [id=0b8ff648531f42dd94998cc01b79bf3e])
I20250623 08:04:44.906973 339 tablet_bootstrap.cc:654] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.907560 2076 tablet_loader.cc:96] loaded metadata for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d])
I20250623 08:04:44.908007 2076 tablet_loader.cc:96] loaded metadata for tablet 38b8cfcbce6a472ab2daae28226cf766 (table default.Table_b161fc2726dc4831988f5ac03028b089 [id=c8effd1e07264a5984e51f380a700652])
I20250623 08:04:44.909001 2076 tablet_loader.cc:96] loaded metadata for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d])
I20250623 08:04:44.909381 2076 tablet_loader.cc:96] loaded metadata for tablet 3c14e325fa5f4f848748c346bc25dde0 (table default.Table_f66b3fa5edc14cffaa95a0a0a4f62f67 [id=ad4cb54e99034dd3af8bdac8fcce4669])
I20250623 08:04:44.910468 2076 tablet_loader.cc:96] loaded metadata for tablet 3cc8503b612d4eb7b8e42d9a3bdfd8f5 (table default.Table_1a60513fe9e94660aa640b5522394bbb [id=79c8424b812d4c98b8dbd8e1fdd5dd52])
I20250623 08:04:44.910964 2076 tablet_loader.cc:96] loaded metadata for tablet 3d7773d6479742f79c38eca34018f94f (table default.Table_25f75684255a4b99a8aff38ee935aac5 [id=e85cbbd0d9db4945930b3fca419e3f2c])
I20250623 08:04:44.911463 2076 tablet_loader.cc:96] loaded metadata for tablet 3fcd0382328f4b0f9ac510c83eb1f68f (table default.Table_e9f2ab1d95fb4341b5d05f6f5bd09f15 [id=f0a8652a629545a9a9bc4c22aba0da74])
I20250623 08:04:44.912309 2076 tablet_loader.cc:96] loaded metadata for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9])
I20250623 08:04:44.912941 2076 tablet_loader.cc:96] loaded metadata for tablet 432d1642c0e44119b5bc05858463a136 (table default.Table_26e019c7e20f450e8fb941854bbdc1e3 [id=69d1f07527404fdda19f124d100f6297])
I20250623 08:04:44.913476 2076 tablet_loader.cc:96] loaded metadata for tablet 45293db3c40b4e4194445f12e7475e66 (table default.Table_e8864c4ee64b4f4dbf04caa321024840 [id=537cece72a774ac99261a4c2a2e52c2e])
I20250623 08:04:44.916466 343 ts_tablet_manager.cc:1397] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.162s user 0.006s sys 0.001s
I20250623 08:04:44.898140 32589 leader_election.cc:304] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:44.918663 343 raft_consensus.cc:357] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.918882 343 raft_consensus.cc:383] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.918934 343 raft_consensus.cc:738] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.919143 2188 raft_consensus.cc:2802] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.919224 2188 raft_consensus.cc:491] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.919312 2188 raft_consensus.cc:3058] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.917789 2013 raft_consensus.cc:491] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.919697 2013 raft_consensus.cc:513] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:44.914891 32453 leader_election.cc:343] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.920290 2013 leader_election.cc:290] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.920337 32453 leader_election.cc:304] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:44.920569 32453 leader_election.cc:304] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:44.920750 1957 raft_consensus.cc:2802] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.920816 1957 raft_consensus.cc:491] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.920869 1957 raft_consensus.cc:3058] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.922058 2188 raft_consensus.cc:513] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.922591 2188 leader_election.cc:290] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.923520 1957 raft_consensus.cc:513] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.924059 1957 leader_election.cc:290] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.924211 1957 raft_consensus.cc:2802] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.924268 1957 raft_consensus.cc:491] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.924327 1957 raft_consensus.cc:3058] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.925552 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7dc5e0fbeba4412aa42d116cd8882238" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.926054 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3136ec837bea4619ae3995ceec9b24d6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.926198 32368 raft_consensus.cc:3058] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.927001 1957 raft_consensus.cc:513] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.927505 1957 leader_election.cc:290] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.928313 1627 consensus_queue.cc:1035] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:44.928798 32368 raft_consensus.cc:2466] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
W20250623 08:04:44.929322 32589 leader_election.cc:343] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.929556 32589 leader_election.cc:304] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:44.929804 2210 raft_consensus.cc:2802] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.929839 2076 tablet_loader.cc:96] loaded metadata for tablet 4597642b2a334baab0a336dace88de0f (table default.Table_cddb0cb673ab4aacb6e69f181369ca8a [id=8afa0fb6b40242fb826b52206a1173c1])
I20250623 08:04:44.929955 2210 raft_consensus.cc:695] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.930239 2210 consensus_queue.cc:237] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.932799 2076 tablet_loader.cc:96] loaded metadata for tablet 4836d621112e46c380c7bc2b7a93117f (table default.Table_afe210848a6b49b2a15d04cf79e1d47c [id=64522ca998ae445696a94c41ca58069a])
I20250623 08:04:44.935220 2076 tablet_loader.cc:96] loaded metadata for tablet 486f00b38c444ea9ba6f6482e05f873d (table default.Table_68f032ade7314d0da985fa8c94b11a8c [id=9895a12eef5a415cacd19cd277721b70])
I20250623 08:04:44.936066 2076 tablet_loader.cc:96] loaded metadata for tablet 48ea78fb01d646d681404009eab9b050 (table default.Table_6b3d1637c4c1443089f04128e190256f [id=3f48141767f441dbbdf5a7742a16176f])
I20250623 08:04:44.938261 2076 tablet_loader.cc:96] loaded metadata for tablet 492ffaeaaa314dcfb7512d8f23946a67 (table default.Table_712680c326284dd782d57dc0ee153879 [id=407f4156ebee45ecb729ad3b3d5f26e5])
I20250623 08:04:44.939179 2188 raft_consensus.cc:491] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.939322 2188 raft_consensus.cc:513] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.939872 2188 leader_election.cc:290] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.940755 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f97f14cbaffb46d681d8b11c0213cae4" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.940949 32368 raft_consensus.cc:2466] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:44.941841 2076 tablet_loader.cc:96] loaded metadata for tablet 4a0615e6f8c346aba5e5f30362bc6de4 (table default.Table_738cccce8e6c47d683b9a7c569aa58e3 [id=20baa502b37c4ec1904f52ddeaf99c34])
I20250623 08:04:44.941884 32589 leader_election.cc:304] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:44.942206 2014 raft_consensus.cc:2802] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:44.942286 2014 raft_consensus.cc:491] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:44.942350 2014 raft_consensus.cc:3058] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.945030 2014 raft_consensus.cc:513] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.945765 2188 raft_consensus.cc:491] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.945873 2188 raft_consensus.cc:513] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.946395 2188 leader_election.cc:290] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:44.946941 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dbbdbebdf3b4b50bda31a94fc0de36f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:44.947705 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f97f14cbaffb46d681d8b11c0213cae4" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.947850 32368 raft_consensus.cc:3058] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.946626 2014 leader_election.cc:290] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:44.947397 32589 leader_election.cc:343] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:44.951568 32368 raft_consensus.cc:2466] T f97f14cbaffb46d681d8b11c0213cae4 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:44.951988 32589 leader_election.cc:304] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:44.952242 2014 raft_consensus.cc:2802] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:44.952370 2014 raft_consensus.cc:695] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:44.952648 2014 consensus_queue.cc:237] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.954885 32381 raft_consensus.cc:3058] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.955456 981 ts_tablet_manager.cc:1918] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250623 08:04:44.958536 2076 tablet_loader.cc:96] loaded metadata for tablet 4b7416ad6a244d978457e529cbc789de (table default.Table_06e661ea03304faaaa16113c730a6d91 [id=60538f85a9b84a1190ef7403747476fb])
I20250623 08:04:44.958982 343 consensus_queue.cc:260] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:44.959874 343 ts_tablet_manager.cc:1428] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20250623 08:04:44.960206 343 tablet_bootstrap.cc:492] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:44.962054 342 tablet_bootstrap.cc:492] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:44.963063 2076 tablet_loader.cc:96] loaded metadata for tablet 4cf1f3943a3d49318c435921ebeaaf33 (table default.Table_39a59f2375e743109b067b2de8b6d4db [id=9125a8d179614084bebc3591092905e5])
I20250623 08:04:44.965586 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55669db7e55042b2bc208ea3218c2d6c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:44.965792 32645 raft_consensus.cc:3058] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.968787 32645 raft_consensus.cc:2466] T 55669db7e55042b2bc208ea3218c2d6c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:44.969797 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 69722us
I20250623 08:04:44.970091 32314 leader_election.cc:304] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:44.970353 2229 raft_consensus.cc:2747] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:44.971583 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c74e8c697d244ac2ac8174bb1fc76e61" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:44.971776 32645 raft_consensus.cc:3058] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.975149 32645 raft_consensus.cc:2466] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:44.956244 981 log.cc:1199] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/ba3d2fa01d574dd992dc904ac024ba3a
I20250623 08:04:44.980520 343 tablet_bootstrap.cc:654] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:44.991155 981 ts_tablet_manager.cc:1939] T ba3d2fa01d574dd992dc904ac024ba3a P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:44.981889 339 tablet_bootstrap.cc:492] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:44.987169 32368 raft_consensus.cc:3058] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.987327 32380 raft_consensus.cc:3058] T 8111e6117c1541ab8905d5a5a7e37292 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:44.987810 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55669db7e55042b2bc208ea3218c2d6c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.991665 339 ts_tablet_manager.cc:1397] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.091s user 0.003s sys 0.000s
I20250623 08:04:44.987872 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c74e8c697d244ac2ac8174bb1fc76e61" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:44.992367 339 raft_consensus.cc:357] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.992596 339 raft_consensus.cc:383] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:44.992681 339 raft_consensus.cc:738] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:44.992938 339 consensus_queue.cc:260] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:44.994385 32314 consensus_peers.cc:487] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6926c49b787941d98c647543bd3b9ed5. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:44.995613 2229 raft_consensus.cc:491] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:44.995743 2229 raft_consensus.cc:513] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:44.996232 2229 leader_election.cc:290] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:44.996313 339 ts_tablet_manager.cc:1428] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250623 08:04:44.996726 339 tablet_bootstrap.cc:492] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:44.998409 339 tablet_bootstrap.cc:654] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.001055 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 361) took 5804 ms. Trace:
I20250623 08:04:45.001184 981 rpcz_store.cc:276] 0623 08:04:39.196929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.202811 (+ 5882us) service_pool.cc:225] Handling call
0623 08:04:44.870308 (+5667497us) ts_tablet_manager.cc:1145] Deleting tablet ba3d2fa01d574dd992dc904ac024ba3a
0623 08:04:44.870553 (+ 245us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:44.903178 (+ 32625us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:44.954555 (+ 51377us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.001040 (+ 46485us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5667275}
I20250623 08:04:45.001778 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 367) took 5690 ms. Trace:
I20250623 08:04:45.001971 981 rpcz_store.cc:276] 0623 08:04:39.311731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.312446 (+ 715us) service_pool.cc:225] Handling call
0623 08:04:45.001618 (+5689172us) ts_tablet_manager.cc:1145] Deleting tablet 1b213c8c685c43868df1e814c79fc61f
0623 08:04:45.001628 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.001764 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5664548}
I20250623 08:04:45.002602 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 368) took 5655 ms. Trace:
I20250623 08:04:45.002795 981 rpcz_store.cc:276] 0623 08:04:39.346810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.346857 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:45.002382 (+5655525us) ts_tablet_manager.cc:1145] Deleting tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159
0623 08:04:45.002440 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.002565 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5618931}
I20250623 08:04:45.003522 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 369) took 5654 ms. Trace:
I20250623 08:04:45.003685 981 rpcz_store.cc:276] 0623 08:04:39.384356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.384555 (+ 199us) service_pool.cc:225] Handling call
0623 08:04:45.003388 (+5618833us) ts_tablet_manager.cc:1145] Deleting tablet f4a0b9402a264800a1ea39707c10da0c
0623 08:04:45.003397 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.003485 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5617505}
I20250623 08:04:45.004122 981 tablet_replica.cc:331] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.004945 2014 raft_consensus.cc:491] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.005103 2014 raft_consensus.cc:513] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.005678 2014 leader_election.cc:290] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.006249 342 ts_tablet_manager.cc:1397] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.159s user 0.003s sys 0.000s
I20250623 08:04:45.006284 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0b1287cf3b6459eafd032afc77c5824" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.006801 32589 leader_election.cc:343] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.008767 2229 raft_consensus.cc:491] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.008908 2229 raft_consensus.cc:513] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.009403 2229 leader_election.cc:290] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.013927 981 raft_consensus.cc:2241] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:44.976729 32453 leader_election.cc:304] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:45.014703 1957 raft_consensus.cc:2802] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:45.014873 1957 raft_consensus.cc:491] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:45.014981 1957 raft_consensus.cc:3058] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:45.015046 2160 consensus_queue.cc:1035] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.015564 981 raft_consensus.cc:2270] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.015942 1627 consensus_queue.cc:1035] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.016194 2160 consensus_queue.cc:1035] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.016269 1627 consensus_queue.cc:1035] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:45.016672 32453 leader_election.cc:343] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.017004 32453 consensus_peers.cc:487] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c9d1c8a67e149478a0d69842a4eb5b2. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:45.017166 32453 leader_election.cc:343] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.017582 32453 consensus_peers.cc:487] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:45.017814 32453 leader_election.cc:343] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.017912 32453 leader_election.cc:304] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.018146 1359 raft_consensus.cc:2747] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:45.018486 32453 consensus_peers.cc:487] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:45.018851 1957 raft_consensus.cc:513] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.019816 1957 leader_election.cc:290] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.020440 2076 tablet_loader.cc:96] loaded metadata for tablet 4e06ef06db2643779f93617e27f100dd (table default.Table_8a32f131cc9048a583663a64ed26a4af [id=e201b6e61b4641b9b4894576ac00fad4])
I20250623 08:04:45.021088 343 tablet_bootstrap.cc:492] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.021263 343 ts_tablet_manager.cc:1397] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.001s
I20250623 08:04:45.021235 342 raft_consensus.cc:357] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.021965 342 raft_consensus.cc:383] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.022115 343 raft_consensus.cc:357] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.022289 343 raft_consensus.cc:383] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.022352 343 raft_consensus.cc:738] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.022380 342 raft_consensus.cc:738] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.022769 342 consensus_queue.cc:260] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.023265 2076 tablet_loader.cc:96] loaded metadata for tablet 4e3920ac85e54379abbcc42dd80db504 (table default.Table_4533668b8daa4ff3ad00ccc95e97e032 [id=08d03b1b488c4b52abebcef8b3d0f853])
I20250623 08:04:45.023197 952 tablet_replica.cc:331] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:45.023829 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34da529ffde4595a23fc6596b30e3e9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.023969 952 raft_consensus.cc:2241] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:45.024037 32519 raft_consensus.cc:2391] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
W20250623 08:04:45.018724 32453 leader_election.cc:343] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.024577 32453 leader_election.cc:343] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.025091 952 raft_consensus.cc:2270] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.025651 342 ts_tablet_manager.cc:1428] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250623 08:04:45.025938 342 tablet_bootstrap.cc:492] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:45.026847 952 ts_tablet_manager.cc:1905] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:45.027175 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 361) could send its response
W20250623 08:04:45.027392 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 367) could send its response
W20250623 08:04:45.027559 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 368) could send its response
W20250623 08:04:45.027707 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 369) could send its response
I20250623 08:04:45.027750 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34da529ffde4595a23fc6596b30e3e9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.028270 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d19d468ce13d4108a9badc36fc503306" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.028919 2229 raft_consensus.cc:491] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.029040 2229 raft_consensus.cc:513] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.029067 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d19d468ce13d4108a9badc36fc503306" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.029191 339 tablet_bootstrap.cc:492] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.029290 339 ts_tablet_manager.cc:1397] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.003s
I20250623 08:04:45.029578 2229 leader_election.cc:290] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.030886 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bfa787f828b411ab68d81bb896f7fb6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.031064 32519 raft_consensus.cc:2391] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:45.031900 981 ts_tablet_manager.cc:1905] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.031116 339 raft_consensus.cc:357] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.032953 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bfa787f828b411ab68d81bb896f7fb6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.033069 339 raft_consensus.cc:383] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.033195 339 raft_consensus.cc:738] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.032078 32314 leader_election.cc:304] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.033695 2229 raft_consensus.cc:3058] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.033879 32314 leader_election.cc:304] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.034186 339 consensus_queue.cc:260] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.034299 2076 tablet_loader.cc:96] loaded metadata for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f])
I20250623 08:04:45.036574 2229 raft_consensus.cc:2747] T a34da529ffde4595a23fc6596b30e3e9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.022610 343 consensus_queue.cc:260] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.037009 2260 raft_consensus.cc:2747] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:45.037366 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c74e8c697d244ac2ac8174bb1fc76e61" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.037552 32513 raft_consensus.cc:2391] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:45.037796 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0107b41d5c04a5c818f6792f2c18ce5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.037927 32518 raft_consensus.cc:3058] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.037458 339 ts_tablet_manager.cc:1428] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250623 08:04:45.039518 339 tablet_bootstrap.cc:492] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.040524 32518 raft_consensus.cc:2466] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.037566 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f957f5575e9475eb9f639e3ad5cb809" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.040817 32514 raft_consensus.cc:3058] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.041184 339 tablet_bootstrap.cc:654] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.038290 32314 leader_election.cc:304] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.042306 1957 consensus_queue.cc:1035] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.042739 2229 raft_consensus.cc:3058] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.043288 32514 raft_consensus.cc:2466] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.043543 343 ts_tablet_manager.cc:1428] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250623 08:04:45.043848 343 tablet_bootstrap.cc:492] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.045254 2229 raft_consensus.cc:2747] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.046149 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3136ec837bea4619ae3995ceec9b24d6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.046304 32512 raft_consensus.cc:2466] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.048668 343 tablet_bootstrap.cc:654] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.054930 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7dc5e0fbeba4412aa42d116cd8882238" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.055153 32512 raft_consensus.cc:2466] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
W20250623 08:04:45.051492 32589 consensus_peers.cc:487] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:45.056689 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3136ec837bea4619ae3995ceec9b24d6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.056860 32512 raft_consensus.cc:3058] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.059973 32512 raft_consensus.cc:2466] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.060930 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dbbdbebdf3b4b50bda31a94fc0de36f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.061157 32512 raft_consensus.cc:2466] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.062533 342 tablet_bootstrap.cc:654] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.062934 2229 raft_consensus.cc:491] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.063134 2229 raft_consensus.cc:513] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.063839 2229 leader_election.cc:290] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.064047 2076 tablet_loader.cc:96] loaded metadata for tablet 51fce46407ac4e57a74ad696e85daac7 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f])
I20250623 08:04:45.064536 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db81250d1e164a7d94affd764fc0ff0c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.064740 32512 raft_consensus.cc:2391] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:45.064692 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db81250d1e164a7d94affd764fc0ff0c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.065274 32314 leader_election.cc:304] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.065809 2229 raft_consensus.cc:3058] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.068313 2229 raft_consensus.cc:2747] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.069895 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0b1287cf3b6459eafd032afc77c5824" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.070056 32514 raft_consensus.cc:2466] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
W20250623 08:04:45.077832 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 359) could send its response
I20250623 08:04:45.078243 32589 leader_election.cc:304] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.078655 2210 raft_consensus.cc:2747] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.079411 32589 leader_election.cc:304] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.079833 2210 raft_consensus.cc:2802] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:45.079958 2210 raft_consensus.cc:695] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:45.080209 2210 consensus_queue.cc:237] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.084218 343 tablet_bootstrap.cc:492] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.084345 343 ts_tablet_manager.cc:1397] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.003s
I20250623 08:04:45.084379 981 ts_tablet_manager.cc:1918] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:45.085122 343 raft_consensus.cc:357] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.085282 343 raft_consensus.cc:383] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.085328 343 raft_consensus.cc:738] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.085170 981 log.cc:1199] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/803d3607d75443568b7ddc9f0c0bdc90
I20250623 08:04:45.085513 343 consensus_queue.cc:260] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.086014 343 ts_tablet_manager.cc:1428] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:45.086670 343 tablet_bootstrap.cc:492] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.088750 343 tablet_bootstrap.cc:654] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.089035 981 ts_tablet_manager.cc:1939] T 803d3607d75443568b7ddc9f0c0bdc90 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:45.105647 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aa398354a9e44633857a75c0bbba08b3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:45.105849 32377 raft_consensus.cc:3058] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:45.105203 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 372) took 5718 ms. Trace:
I20250623 08:04:45.106807 2076 tablet_loader.cc:96] loaded metadata for tablet 540314ce8e3a44c1afb43a1e60b07ca8 (table default.Table_47c06fdd0628467d8a0727b9adeb890d [id=4aea4a71b2e64e81bf9cca2a06f15521])
I20250623 08:04:45.107314 2076 tablet_loader.cc:96] loaded metadata for tablet 5443f12974bf41929d9cd937f146cd56 (table default.Table_eda413af2cf84c009c8fc4b375b57e2a [id=898ec01859bf4ba29e38b4eaf3d5392e])
I20250623 08:04:45.106125 981 rpcz_store.cc:276] 0623 08:04:39.386124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.386445 (+ 321us) service_pool.cc:225] Handling call
0623 08:04:45.004058 (+5617613us) ts_tablet_manager.cc:1145] Deleting tablet 803d3607d75443568b7ddc9f0c0bdc90
0623 08:04:45.004089 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.051944 (+ 47855us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.083435 (+ 31491us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.105018 (+ 21583us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8860,"tablet-delete.queue_time_us":5617337}
I20250623 08:04:45.107959 2076 tablet_loader.cc:96] loaded metadata for tablet 55669db7e55042b2bc208ea3218c2d6c (table default.Table_1476eae1111b4f1dbd7c0f5dd0dd9b34 [id=b005275508b748c7a8494ffebdc0bd49])
I20250623 08:04:45.108520 32377 raft_consensus.cc:2466] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 2.
I20250623 08:04:45.108664 2076 tablet_loader.cc:96] loaded metadata for tablet 57cd427592714247a2e075881f9827fb (table default.Table_f7811b4c393d498cbf95abe8a8cdcd38 [id=1c335fc6495f4256a2d7e1f4fd66e875])
I20250623 08:04:45.109303 2076 tablet_loader.cc:96] loaded metadata for tablet 5930085e18d84823b27791a65750663f (table default.Table_1631659a168049fba9411207edd8b56f [id=10d3dacba1db466893ef47b1c42a8f15])
I20250623 08:04:45.109699 2076 tablet_loader.cc:96] loaded metadata for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (table default.Table_b764664d1bdc4441907690b6616f3570 [id=ad7210d2665f463c8591e79cf62d15dc])
I20250623 08:04:45.110100 2076 tablet_loader.cc:96] loaded metadata for tablet 5b1a601763e44934a7db2f53d05e43a5 (table default.Table_0666030ad7ec4ef89335cdbaf0cca443 [id=bf6ca5aa8d5044868244a9b37cfa9519])
I20250623 08:04:45.110070 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 370) took 5725 ms. Trace:
I20250623 08:04:45.110967 981 rpcz_store.cc:276] 0623 08:04:39.384487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.389846 (+ 5359us) service_pool.cc:225] Handling call
0623 08:04:45.109149 (+5719303us) ts_tablet_manager.cc:1145] Deleting tablet cdec6e9573b041e0a57d1e0e0239d97f
0623 08:04:45.109164 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.109866 (+ 702us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5718934}
W20250623 08:04:45.113619 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 372) could send its response
I20250623 08:04:45.114511 342 tablet_bootstrap.cc:492] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:45.114531 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 374) took 5664 ms. Trace:
I20250623 08:04:45.118034 981 rpcz_store.cc:276] 0623 08:04:39.449781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.449841 (+ 60us) service_pool.cc:225] Handling call
0623 08:04:45.113372 (+5663531us) ts_tablet_manager.cc:1145] Deleting tablet 90ceb0f8d5c94c4e9427042d8a35e16c
0623 08:04:45.113843 (+ 471us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.114363 (+ 520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5663332}
W20250623 08:04:45.118247 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 370) could send its response
I20250623 08:04:45.118616 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 376) took 5281 ms. Trace:
I20250623 08:04:45.118850 981 rpcz_store.cc:276] 0623 08:04:39.837170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.837223 (+ 53us) service_pool.cc:225] Handling call
0623 08:04:45.118464 (+5281241us) ts_tablet_manager.cc:1145] Deleting tablet aface2a39d764a6cb6e4965867ab7b38
0623 08:04:45.118475 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.118600 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5281017}
I20250623 08:04:45.119982 2076 tablet_loader.cc:96] loaded metadata for tablet 5e4678ca1c8f498c83c6bc2553882120 (table default.Table_e14bd6f5e20e42b1898bce238d4d986c [id=3a0c95fd4bd84b80ad1f5951cbef92e8])
I20250623 08:04:45.120414 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 377) took 5283 ms. Trace:
I20250623 08:04:45.121156 2076 tablet_loader.cc:96] loaded metadata for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352])
I20250623 08:04:45.121569 2076 tablet_loader.cc:96] loaded metadata for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc])
I20250623 08:04:45.122215 2076 tablet_loader.cc:96] loaded metadata for tablet 623e64e1cd7d4a458fb30ceece2989fd (table default.Table_8e6f4c63178440f4ba5c9f65d213df3d [id=3d71395c0fab47b68b1dcc47bfdc46fb])
I20250623 08:04:45.121096 981 rpcz_store.cc:276] 0623 08:04:39.837301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.837343 (+ 42us) service_pool.cc:225] Handling call
0623 08:04:45.119519 (+5282176us) ts_tablet_manager.cc:1145] Deleting tablet 1aa604f500984e3b969e15592cdbed6f
0623 08:04:45.119531 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.120218 (+ 687us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5281775}
I20250623 08:04:45.123126 2076 tablet_loader.cc:96] loaded metadata for tablet 6857437f85ab40f6a24e74ce3b5efbdc (table default.Table_579c3f68f43e43b790fc83f17ad79191 [id=5bc2455d8b674ada83ce98eedb5f8873])
I20250623 08:04:45.124173 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 378) took 5219 ms. Trace:
I20250623 08:04:45.124799 981 rpcz_store.cc:276] 0623 08:04:39.904433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.904508 (+ 75us) service_pool.cc:225] Handling call
0623 08:04:45.123364 (+5218856us) ts_tablet_manager.cc:1145] Deleting tablet 185c2bf7b9254947bb76b7fe0924559e
0623 08:04:45.123380 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.124155 (+ 775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5185199}
I20250623 08:04:45.126741 343 tablet_bootstrap.cc:492] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.126943 343 ts_tablet_manager.cc:1397] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.002s
I20250623 08:04:45.127873 342 ts_tablet_manager.cc:1397] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.000s
I20250623 08:04:45.127744 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 379) took 5172 ms. Trace:
I20250623 08:04:45.128010 343 raft_consensus.cc:357] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.128206 343 raft_consensus.cc:383] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.128288 2188 raft_consensus.cc:491] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.128290 343 raft_consensus.cc:738] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.128417 2188 raft_consensus.cc:513] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.128602 343 consensus_queue.cc:260] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.129024 2188 leader_election.cc:290] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.128314 981 rpcz_store.cc:276] 0623 08:04:39.954733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.954813 (+ 80us) service_pool.cc:225] Handling call
0623 08:04:45.126780 (+5171967us) ts_tablet_manager.cc:1145] Deleting tablet 9dbbdbebdf3b4b50bda31a94fc0de36f
0623 08:04:45.126795 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.127381 (+ 586us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5171395}
I20250623 08:04:45.129330 343 ts_tablet_manager.cc:1428] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:45.129609 343 tablet_bootstrap.cc:492] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.129966 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6857437f85ab40f6a24e74ce3b5efbdc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.130869 32589 leader_election.cc:343] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.130940 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 380) took 5173 ms. Trace:
I20250623 08:04:45.132171 32589 leader_election.cc:304] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.132640 2188 raft_consensus.cc:2802] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:45.131490 981 rpcz_store.cc:276] 0623 08:04:39.957469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.957521 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:45.130043 (+5172522us) ts_tablet_manager.cc:1145] Deleting tablet 30dfa986dfd44658a0ffb07796030520
0623 08:04:45.130053 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.130921 (+ 868us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5172179}
I20250623 08:04:45.132766 2188 raft_consensus.cc:695] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:45.133139 2188 consensus_queue.cc:237] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.134097 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 401) took 4496 ms. Trace:
I20250623 08:04:45.134804 981 rpcz_store.cc:276] 0623 08:04:40.637355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.637419 (+ 64us) service_pool.cc:225] Handling call
0623 08:04:45.133322 (+4495903us) ts_tablet_manager.cc:1145] Deleting tablet a383d10d53b04d5699dd3c4bad8878c7
0623 08:04:45.133333 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.134081 (+ 748us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4495491}
I20250623 08:04:45.136837 981 tablet_replica.cc:331] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.142553 2076 tablet_loader.cc:96] loaded metadata for tablet 6c33fa0a7ff34d2985c5e07fcf88eaa1 (table default.Table_fa02183517ad4115af0d9fd8ce2cd5d6 [id=1d1ad6c8b80243d08a0d97ee5f1331d5])
I20250623 08:04:45.142730 343 tablet_bootstrap.cc:654] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:45.148634 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 374) could send its response
I20250623 08:04:45.149144 2076 tablet_loader.cc:96] loaded metadata for tablet 6f9e0d1b85a94ae9ae86e53688a55db5 (table default.Table_464d0877823d4ed399bfd7caf198511d [id=73ffec23a77441678c2f9d76087ba5f0])
W20250623 08:04:45.149204 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 376) could send its response
W20250623 08:04:45.149343 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 377) could send its response
W20250623 08:04:45.149456 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 378) could send its response
W20250623 08:04:45.149549 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 379) could send its response
W20250623 08:04:45.149647 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 380) could send its response
W20250623 08:04:45.149748 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 401) could send its response
I20250623 08:04:45.149973 2229 raft_consensus.cc:491] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.150084 2076 tablet_loader.cc:96] loaded metadata for tablet 712abfca1fa4489f882534c33b9db1be (table default.Table_e5d470e2745643ee95ca5c01126dee26 [id=836b8d720fae4622a556600a1a9c1d91])
I20250623 08:04:45.150120 2229 raft_consensus.cc:513] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.150498 2076 tablet_loader.cc:96] loaded metadata for tablet 71fadb10e6134478b013358bba056389 (table default.Table_4a19c65595654fa98efc42626b6a9d85 [id=2979e4296458439ea5753b6ba67634ac])
I20250623 08:04:45.150777 2229 leader_election.cc:290] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.152508 2076 tablet_loader.cc:96] loaded metadata for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b])
I20250623 08:04:45.152930 32589 leader_election.cc:304] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.153208 2014 raft_consensus.cc:2802] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.153294 2014 raft_consensus.cc:491] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.153379 2014 raft_consensus.cc:3058] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.155959 343 tablet_bootstrap.cc:492] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.158667 339 tablet_bootstrap.cc:492] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.158916 339 ts_tablet_manager.cc:1397] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.119s user 0.009s sys 0.008s
I20250623 08:04:45.159837 339 raft_consensus.cc:357] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.160073 339 raft_consensus.cc:383] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.160212 339 raft_consensus.cc:738] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.160630 339 consensus_queue.cc:260] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.156381 2014 raft_consensus.cc:513] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.161520 343 ts_tablet_manager.cc:1397] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.003s
I20250623 08:04:45.161927 2014 leader_election.cc:290] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.162001 339 ts_tablet_manager.cc:1428] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:45.162274 343 raft_consensus.cc:357] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.162437 343 raft_consensus.cc:383] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.162492 343 raft_consensus.cc:738] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.162551 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7dc5e0fbeba4412aa42d116cd8882238" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:45.162673 343 consensus_queue.cc:260] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:45.163071 32589 leader_election.cc:343] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.163877 343 ts_tablet_manager.cc:1428] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:45.164134 343 tablet_bootstrap.cc:492] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.164333 339 tablet_bootstrap.cc:492] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.169730 2076 tablet_loader.cc:96] loaded metadata for tablet 736eb56dae194df2af90bf33d3294a4e (table default.Table_4e8bf0e3037248cfb48d25ab98e5468c [id=cb2c6ba697ca494c8eb60b65624a0517])
I20250623 08:04:45.170428 2076 tablet_loader.cc:96] loaded metadata for tablet 7419cb91360f49e1b3cb7e90732558c5 (table default.Table_161ad5ddd1cb40e0989e5c237a205db1 [id=7cc0919082d04fc4a02aa856a42e5529])
I20250623 08:04:45.170850 2076 tablet_loader.cc:96] loaded metadata for tablet 74842fde5152419ca35f3950e465247a (table default.Table_9aeb558e2ae64fe2b45bef5d2b6a2625 [id=2bfacaaab9f347b1a77e5eb29a9c485e])
I20250623 08:04:45.171540 2076 tablet_loader.cc:96] loaded metadata for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491])
I20250623 08:04:45.172413 2076 tablet_loader.cc:96] loaded metadata for tablet 764244f33a1a48fdad410597429d3461 (table default.Table_1e293608b61743279ff20716c2953e24 [id=d7d9ed45091446fea8933157fd6e0db7])
I20250623 08:04:45.173138 2076 tablet_loader.cc:96] loaded metadata for tablet 776b412b3f564a7581c6a5e1c84e8499 (table default.Table_2ae738ad8d0d430981d05bf2d7239d73 [id=491fdbceb3d448d3904beb75d0881652])
I20250623 08:04:45.173192 339 tablet_bootstrap.cc:654] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.173556 2076 tablet_loader.cc:96] loaded metadata for tablet 7874a1d5cb9a464294ca32b2767efbf3 (table default.Table_8ec723069381477ca64f5aaad0df76fb [id=2a5a01b05a554ac1ad6abd21ad00d734])
I20250623 08:04:45.174300 2076 tablet_loader.cc:96] loaded metadata for tablet 79014324599840038cf53ca02cbc6f7e (table default.Table_ff6e3cec929a433b8ac68152b9095dfc [id=ce7434eee7664a8686eb22f28df7f1d8])
W20250623 08:04:45.174386 32589 consensus_peers.cc:487] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:45.175344 32589 leader_election.cc:343] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: f97f14cbaffb46d681d8b11c0213cae4
I20250623 08:04:45.175508 343 tablet_bootstrap.cc:654] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.177995 342 raft_consensus.cc:357] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.178328 342 raft_consensus.cc:383] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.178570 342 raft_consensus.cc:738] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.176035 32589 leader_election.cc:304] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.179145 2014 raft_consensus.cc:2802] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.179296 2014 raft_consensus.cc:491] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.179422 2014 raft_consensus.cc:3058] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.182845 2014 raft_consensus.cc:513] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.183688 2014 leader_election.cc:290] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.186434 952 ts_tablet_manager.cc:1918] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:45.186915 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dbbdbebdf3b4b50bda31a94fc0de36f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:45.187476 32589 leader_election.cc:343] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.188474 2076 tablet_loader.cc:96] loaded metadata for tablet 7adec1482119405daca4aa603770ae22 (table default.Table_d94ea6a713d546fa80c002017bd9a39f [id=1e194b3160b64acabac38acfd50a64b3])
I20250623 08:04:45.189932 2076 tablet_loader.cc:96] loaded metadata for tablet 7dc5e0fbeba4412aa42d116cd8882238 (table default.Table_ea4d110c161f4b008acde5f22ffa3562 [id=79d117a74f18466ab87a1701e641e7da])
I20250623 08:04:45.192814 343 tablet_bootstrap.cc:492] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.193069 343 ts_tablet_manager.cc:1397] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250623 08:04:45.194489 343 raft_consensus.cc:357] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.194844 343 raft_consensus.cc:383] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.194996 343 raft_consensus.cc:738] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.195446 343 consensus_queue.cc:260] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.196611 343 ts_tablet_manager.cc:1428] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:45.197177 343 tablet_bootstrap.cc:492] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.197899 2076 tablet_loader.cc:96] loaded metadata for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 (table default.Table_2ecc70ddf2dc40e081fefb676d46a68d [id=3a1a4d6d2cc24dc8abe85a4597c0d0c8])
I20250623 08:04:45.198391 2076 tablet_loader.cc:96] loaded metadata for tablet 7f416cdb596a4ed5a302acd07bb84b9c (table default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e])
I20250623 08:04:45.199841 339 tablet_bootstrap.cc:492] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.200028 339 ts_tablet_manager.cc:1397] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.003s
I20250623 08:04:45.200506 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "affa3da7540648d28c5f00a3cf348eb1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.200999 339 raft_consensus.cc:357] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.201319 339 raft_consensus.cc:383] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.201457 339 raft_consensus.cc:738] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.201714 981 raft_consensus.cc:2241] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.201902 339 consensus_queue.cc:260] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.202128 981 pending_rounds.cc:70] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:45.202822 981 pending_rounds.cc:77] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727459694399488 op_type: NO_OP noop_request { }
I20250623 08:04:45.202862 339 ts_tablet_manager.cc:1428] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:45.202922 981 raft_consensus.cc:2887] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:45.202998 981 raft_consensus.cc:2270] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.203362 339 tablet_bootstrap.cc:492] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.205428 2076 tablet_loader.cc:96] loaded metadata for tablet 7f890ee0747b4df0b82bc3b396191124 (table default.Table_7ccb5edd6b824f308d349c51d926d963 [id=9f97f6deca194137a353ea221d0e3676])
I20250623 08:04:45.207211 981 ts_tablet_manager.cc:1905] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.212558 952 log.cc:1199] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/803d3607d75443568b7ddc9f0c0bdc90
I20250623 08:04:45.188964 2160 consensus_queue.cc:1035] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.198633 1957 raft_consensus.cc:491] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.215759 1957 raft_consensus.cc:513] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.216356 1957 leader_election.cc:290] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.199288 32453 leader_election.cc:304] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.199862 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "affa3da7540648d28c5f00a3cf348eb1" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.217063 1957 raft_consensus.cc:2802] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:45.217168 1957 raft_consensus.cc:695] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:45.217115 343 tablet_bootstrap.cc:654] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.217398 1957 consensus_queue.cc:237] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.217727 981 ts_tablet_manager.cc:1918] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:45.217834 981 log.cc:1199] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/4b11777e0bd04e61a5bc8bb6ec4d97c3
I20250623 08:04:45.218261 981 ts_tablet_manager.cc:1939] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:45.219133 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 402) took 4425 ms. Trace:
I20250623 08:04:45.219287 981 rpcz_store.cc:276] 0623 08:04:40.794104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.794174 (+ 70us) service_pool.cc:225] Handling call
0623 08:04:45.136414 (+4342240us) ts_tablet_manager.cc:1145] Deleting tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3
0623 08:04:45.136425 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.202049 (+ 65624us) spinlock_profiling.cc:243] Waited 63.6 ms on lock 0x616000291454. stack: 00007f7f6e5c4d56 00007f7f6e5c4b10 00007f7f6d5bb29d 0000000000970a84 0000000000970a4d 000000000097041a 00007f7f7a022e80 00007f7f7b334a93 00007f7f7de10c97 00007f7f7de42d24 00007f7f7de24780 00007f7f7de2450d 0000000000a751bc 00007f7f6e60a3ff 00007f7f6e60d6cb 00007f7f6e60d548
0623 08:04:45.210399 (+ 8350us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.217608 (+ 7209us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.219112 (+ 1504us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146282112,"tablet-delete.queue_time_us":4339324}
I20250623 08:04:45.219795 2076 tablet_loader.cc:96] loaded metadata for tablet 8111e6117c1541ab8905d5a5a7e37292 (table default.Table_6b3f6aad5faf4f2a88573140691c6291 [id=8e322f09172b40a8ba5aea16337fb020])
I20250623 08:04:45.219894 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 403) took 4386 ms. Trace:
I20250623 08:04:45.221251 981 rpcz_store.cc:276] 0623 08:04:40.833252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.833306 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:45.219739 (+4386433us) ts_tablet_manager.cc:1145] Deleting tablet d5cd316e6f4447be958611ab07ed31fc
0623 08:04:45.219752 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.219875 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4386220}
I20250623 08:04:45.221181 339 tablet_bootstrap.cc:654] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.221578 2076 tablet_loader.cc:96] loaded metadata for tablet 8145567741f34d9aa70c5b7b86f7e1c6 (table default.Table_412c3eb097714d94bf0b6c487f1c3caa [id=bbf01747113a4685a0cd746091a9e5bc])
I20250623 08:04:45.222371 2076 tablet_loader.cc:96] loaded metadata for tablet 83e8e9739af0464787afc3580a034cd6 (table default.Table_43ff9ceffb4042b5ab1b041b76bdfc96 [id=2013d4281dc84c089dc17304aefd0837])
I20250623 08:04:45.221617 981 tablet_replica.cc:331] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.222843 981 raft_consensus.cc:2241] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.223048 981 raft_consensus.cc:2270] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.223194 2076 tablet_loader.cc:96] loaded metadata for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae])
I20250623 08:04:45.223723 981 ts_tablet_manager.cc:1905] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.188525 342 consensus_queue.cc:260] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.214957 1627 raft_consensus.cc:491] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.224368 1627 raft_consensus.cc:513] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.224458 2076 tablet_loader.cc:96] loaded metadata for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451])
I20250623 08:04:45.224854 1627 leader_election.cc:290] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.224846 2076 tablet_loader.cc:96] loaded metadata for tablet 8911308855cb4414be426dbcec592f4a (table default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22])
I20250623 08:04:45.225786 342 ts_tablet_manager.cc:1428] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.098s user 0.002s sys 0.000s
I20250623 08:04:45.226912 342 tablet_bootstrap.cc:492] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:45.227761 32453 leader_election.cc:304] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.230268 1627 raft_consensus.cc:2802] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:45.230378 1627 raft_consensus.cc:695] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:45.230602 1627 consensus_queue.cc:237] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.230887 2229 raft_consensus.cc:491] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.231015 2229 raft_consensus.cc:513] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.231567 2229 leader_election.cc:290] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.235997 339 tablet_bootstrap.cc:492] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.236162 339 ts_tablet_manager.cc:1397] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.002s
I20250623 08:04:45.236954 339 raft_consensus.cc:357] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.237186 2076 tablet_loader.cc:96] loaded metadata for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287])
I20250623 08:04:45.238405 2076 tablet_loader.cc:96] loaded metadata for tablet 8da872257b26441c8456446426affe7a (table default.Table_b91721eeaabb4377b4d62483ad065bee [id=8544c855ea7e4fed887765c7730c3ccd])
I20250623 08:04:45.239133 2076 tablet_loader.cc:96] loaded metadata for tablet 8e17e1ca404b462da43aa1db7dc3b556 (table default.Table_6a8ea07f0b174864b81addf035ed1f34 [id=dfff0b62abf54558b0b04a3b5d8cd896])
I20250623 08:04:45.239548 2076 tablet_loader.cc:96] loaded metadata for tablet 8e2a562010944837b4e486e0facd47cf (table default.Table_334749aabade478fbbc2badb1e4960bf [id=ec79bae33a55429fb05f25780369b398])
I20250623 08:04:45.239974 2076 tablet_loader.cc:96] loaded metadata for tablet 8f83a6d453ec4ffe85282b4385df5f11 (table default.Table_7359680519754448a8464b0e8131bd55 [id=0795852ef28a4989888d3ac032da65d5])
I20250623 08:04:45.240381 2076 tablet_loader.cc:96] loaded metadata for tablet 8f8f8a132e6245f9a04b5bb6f2253c76 (table default.Table_c748f600d56f4374bdb31ad502ec7557 [id=3b342fcf5bff4bf0bb19c90fca9015c3])
I20250623 08:04:45.241282 2076 tablet_loader.cc:96] loaded metadata for tablet 90037c523a1f4118af268e088bb97062 (table default.Table_5cf87169ed0d4987b396195d518aa055 [id=5edc67ced51e4cde9dbfcf448d24b87d])
I20250623 08:04:45.242931 2076 tablet_loader.cc:96] loaded metadata for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99])
I20250623 08:04:45.243772 343 tablet_bootstrap.cc:492] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.244009 343 ts_tablet_manager.cc:1397] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.000s
I20250623 08:04:45.244885 343 raft_consensus.cc:357] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.245124 343 raft_consensus.cc:383] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.245260 343 raft_consensus.cc:738] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.245662 343 consensus_queue.cc:260] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.237283 981 ts_tablet_manager.cc:1918] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:45.247143 981 log.cc:1199] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/aa398354a9e44633857a75c0bbba08b3
I20250623 08:04:45.247087 339 raft_consensus.cc:383] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.247287 339 raft_consensus.cc:738] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.247359 2188 raft_consensus.cc:491] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.247205 343 ts_tablet_manager.cc:1428] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:45.247576 981 ts_tablet_manager.cc:1939] T aa398354a9e44633857a75c0bbba08b3 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:45.247504 2188 raft_consensus.cc:513] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.247501 339 consensus_queue.cc:260] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.248733 2188 leader_election.cc:290] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.248808 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 404) took 4020 ms. Trace:
I20250623 08:04:45.248891 981 rpcz_store.cc:276] 0623 08:04:41.228417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.228463 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:45.221575 (+3993112us) ts_tablet_manager.cc:1145] Deleting tablet aa398354a9e44633857a75c0bbba08b3
0623 08:04:45.221583 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.227003 (+ 5420us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.237169 (+ 10166us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.248769 (+ 11600us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":439,"tablet-delete.queue_time_us":3992923}
I20250623 08:04:45.248977 343 tablet_bootstrap.cc:492] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.249284 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 405) took 4020 ms. Trace:
I20250623 08:04:45.249318 339 ts_tablet_manager.cc:1428] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250623 08:04:45.249351 981 rpcz_store.cc:276] 0623 08:04:41.228773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.228812 (+ 39us) service_pool.cc:225] Handling call
0623 08:04:45.249159 (+4020347us) ts_tablet_manager.cc:1145] Deleting tablet 74c1cdc83ffd4a09a163479ac58fd405
0623 08:04:45.249167 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.249268 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4020190}
I20250623 08:04:45.249567 339 tablet_bootstrap.cc:492] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.249593 981 tablet_replica.cc:331] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.249754 981 raft_consensus.cc:2241] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.249886 981 raft_consensus.cc:2270] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.250361 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b61b02cf2904404f8c0ffef7d268fefc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.251200 32589 leader_election.cc:343] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.251322 981 ts_tablet_manager.cc:1905] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.260267 2076 tablet_loader.cc:96] loaded metadata for tablet 9bb99da3b8fa4dc6aa7f11a829d3f2d2 (table default.Table_3980cd736bd342f4bcd5cf289c1c8c25 [id=47a9c2f4140743c3a357032721181f32])
W20250623 08:04:45.263482 32453 leader_election.cc:343] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: aa398354a9e44633857a75c0bbba08b3
I20250623 08:04:45.264693 2076 tablet_loader.cc:96] loaded metadata for tablet a0b1287cf3b6459eafd032afc77c5824 (table default.Table_cc3b284f368e4123bf1a2da98f4f4fa8 [id=906debc01a6f45a49a86b5f939ac1dee])
I20250623 08:04:45.265707 343 tablet_bootstrap.cc:654] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.266837 342 tablet_bootstrap.cc:654] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.267241 339 tablet_bootstrap.cc:654] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.267550 2188 consensus_queue.cc:1035] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.267663 981 ts_tablet_manager.cc:1918] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:45.267913 981 log.cc:1199] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/3136ec837bea4619ae3995ceec9b24d6
I20250623 08:04:45.268782 952 ts_tablet_manager.cc:1939] T 803d3607d75443568b7ddc9f0c0bdc90 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:45.268311 981 ts_tablet_manager.cc:1939] T 3136ec837bea4619ae3995ceec9b24d6 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:45.269625 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 362) took 5852 ms. Trace:
W20250623 08:04:45.269835 32314 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 54636.6us
W20250623 08:04:45.270085 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 402) could send its response
I20250623 08:04:45.270324 2076 tablet_loader.cc:96] loaded metadata for tablet a0e9b2f810014e81a29ca9b41cd3f8ba (table default.Table_3115dd6f7dc948aa927f4ab714867992 [id=e19b2510cd0b45c5bb9db6e00b3bc24e])
W20250623 08:04:45.270397 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 403) could send its response
I20250623 08:04:45.269901 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 406) took 3959 ms. Trace:
I20250623 08:04:45.270819 2076 tablet_loader.cc:96] loaded metadata for tablet a1c82e4d2db64026a04c5c84bc1e09ec (table default.Table_1fe10162eb7047d0a2ec4c517bb9fc7f [id=31a4aa2ab5e0455db4b94ea79d38f524])
I20250623 08:04:45.270717 981 rpcz_store.cc:276] 0623 08:04:41.310598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.310812 (+ 214us) service_pool.cc:225] Handling call
0623 08:04:45.249550 (+3938738us) ts_tablet_manager.cc:1145] Deleting tablet 3136ec837bea4619ae3995ceec9b24d6
0623 08:04:45.249560 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.255335 (+ 5775us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.267553 (+ 12218us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.269884 (+ 2331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3938591}
I20250623 08:04:45.271076 981 tablet_replica.cc:331] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.271281 981 raft_consensus.cc:2241] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.271456 981 raft_consensus.cc:2270] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.271499 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b04e38d3da44301a450fe1fb5e2ada6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:45.272706 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 404) could send its response
W20250623 08:04:45.273247 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 405) could send its response
I20250623 08:04:45.273590 2076 tablet_loader.cc:96] loaded metadata for tablet a1cf375214c54f3491098db07faedd9f (table default.Table_56c6dd0a0ee64ed088889c23db0fd2e9 [id=4199d682d171458ab79e4a36807c7f6b])
I20250623 08:04:45.273013 981 ts_tablet_manager.cc:1905] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.274369 2076 tablet_loader.cc:96] loaded metadata for tablet a29c7fd219604bf39f40ccd09dedd2d2 (table default.Table_98f40c99d9ff4819b958cd84cc5dc3a9 [id=13cf327a200c490992163a70f5a275cd])
I20250623 08:04:45.274511 32314 leader_election.cc:304] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.280156 2076 tablet_loader.cc:96] loaded metadata for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675])
I20250623 08:04:45.280691 2076 tablet_loader.cc:96] loaded metadata for tablet a36c01f33f4b423aba86bd9a7f212c5d (table default.Table_87e0e2fcec324a53ae2cfd5b3156e071 [id=75e5879d4117409da29206bb682898cc])
I20250623 08:04:45.281157 2076 tablet_loader.cc:96] loaded metadata for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267])
I20250623 08:04:45.282166 343 tablet_bootstrap.cc:492] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.286870 343 ts_tablet_manager.cc:1397] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.002s
I20250623 08:04:45.288179 343 raft_consensus.cc:357] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.295143 2076 tablet_loader.cc:96] loaded metadata for tablet a38eecf4a94d4536b98d4f6e037a0094 (table default.Table_e216bbb5ac2a4720b36183e30bd6ed91 [id=57e548bc818e4968b432cc3af298e675])
I20250623 08:04:45.295827 2076 tablet_loader.cc:96] loaded metadata for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be])
I20250623 08:04:45.295984 343 raft_consensus.cc:383] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.296105 343 raft_consensus.cc:738] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.297380 2188 raft_consensus.cc:491] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.297717 2076 tablet_loader.cc:96] loaded metadata for tablet a79238b6f6e74ae89c52eefb95190264 (table default.Table_5d6ec4f6805b4e0883b8d0419625d033 [id=545b478879aa4f4cb35393e3065d3ff7])
I20250623 08:04:45.297676 2188 raft_consensus.cc:513] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.297426 343 consensus_queue.cc:260] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.298071 2014 raft_consensus.cc:491] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.298504 2014 raft_consensus.cc:513] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.298978 343 ts_tablet_manager.cc:1428] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250623 08:04:45.300185 2014 leader_election.cc:290] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.300536 2188 leader_election.cc:290] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.301260 339 tablet_bootstrap.cc:492] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.301491 339 ts_tablet_manager.cc:1397] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.003s
I20250623 08:04:45.301523 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf1f3943a3d49318c435921ebeaaf33" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:45.301869 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cca63991d5b4ef5bd8a9181d127ebe3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:45.290524 981 ts_tablet_manager.cc:1918] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250623 08:04:45.302390 32589 leader_election.cc:343] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.270998 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b04e38d3da44301a450fe1fb5e2ada6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:45.302590 32589 leader_election.cc:343] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.302697 32512 raft_consensus.cc:2391] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:45.302649 981 log.cc:1199] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/16331ba6a34942a5a1b4ff4ac7bac55e
I20250623 08:04:45.303001 952 rpcz_store.cc:276] 0623 08:04:39.417845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.418057 (+ 212us) service_pool.cc:225] Handling call
0623 08:04:45.023139 (+5605082us) ts_tablet_manager.cc:1145] Deleting tablet 803d3607d75443568b7ddc9f0c0bdc90
0623 08:04:45.023152 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.115495 (+ 92343us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.186312 (+ 70817us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.269606 (+ 83294us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":784,"tablet-delete.queue_time_us":5604837}
I20250623 08:04:45.303450 343 tablet_bootstrap.cc:492] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.303541 952 tablet_replica.cc:331] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:45.303779 952 raft_consensus.cc:2241] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.303961 952 raft_consensus.cc:2270] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.303817 981 ts_tablet_manager.cc:1939] T 16331ba6a34942a5a1b4ff4ac7bac55e P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:45.304507 952 ts_tablet_manager.cc:1905] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.305181 343 tablet_bootstrap.cc:654] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.297226 32453 leader_election.cc:304] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:45.308070 1953 raft_consensus.cc:2802] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:45.308230 1953 raft_consensus.cc:695] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:45.308640 2076 tablet_loader.cc:96] loaded metadata for tablet affa3da7540648d28c5f00a3cf348eb1 (table default.Table_9c6bc2fbfc274d2e8414205f629cb439 [id=42f65c762e2b46469d3ff189b698b447])
I20250623 08:04:45.308632 1953 consensus_queue.cc:237] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.309032 2076 tablet_loader.cc:96] loaded metadata for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5])
I20250623 08:04:45.309633 2076 tablet_loader.cc:96] loaded metadata for tablet b13dfc9afcbf43c685ca49d5ae0c5560 (table default.Table_9a49c84f75b84594ab81fcf31da3991f [id=7b34109f27f5441e855b23fb253a8e99])
I20250623 08:04:45.310002 2076 tablet_loader.cc:96] loaded metadata for tablet b19e5f2359a84c7ea02a0b25240a54cb (table default.Table_640be163e2114433bd7079aa3a602116 [id=d422f6202d1842e998a23ec064757334])
I20250623 08:04:45.302340 342 tablet_bootstrap.cc:492] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:45.315008 342 ts_tablet_manager.cc:1397] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.000s
I20250623 08:04:45.315769 342 raft_consensus.cc:357] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.315922 342 raft_consensus.cc:383] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.315970 342 raft_consensus.cc:738] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.316130 342 consensus_queue.cc:260] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.316603 342 ts_tablet_manager.cc:1428] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:45.316859 342 tablet_bootstrap.cc:492] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:45.317569 2076 tablet_loader.cc:96] loaded metadata for tablet b300341952d34600ba745245f51482d4 (table default.Table_79fce459ec304a14823e0a2179f6b804 [id=57b3f76fe53a4f69bfb544ad72a7ca52])
I20250623 08:04:45.324036 2229 raft_consensus.cc:2747] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.324695 339 raft_consensus.cc:357] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:45.324797 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 50309.9us
I20250623 08:04:45.324857 339 raft_consensus.cc:383] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.324906 339 raft_consensus.cc:738] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:45.324928 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 406) could send its response
I20250623 08:04:45.325105 339 consensus_queue.cc:260] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.325675 2076 tablet_loader.cc:96] loaded metadata for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181])
I20250623 08:04:45.326215 339 ts_tablet_manager.cc:1428] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250623 08:04:45.326660 339 tablet_bootstrap.cc:492] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.326820 32589 leader_election.cc:304] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.327230 2188 raft_consensus.cc:2802] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.327330 2188 raft_consensus.cc:491] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.327419 2188 raft_consensus.cc:3058] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.328137 2076 tablet_loader.cc:96] loaded metadata for tablet b91d3650e56441189c3cb74b01cd7fa7 (table default.Table_1ced320c61ec4355bb2686a5c84a55eb [id=61c080721215406397c00aff27538c61])
I20250623 08:04:45.329262 339 tablet_bootstrap.cc:654] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.329226 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 410) took 3433 ms. Trace:
I20250623 08:04:45.330096 2188 raft_consensus.cc:513] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.330228 981 rpcz_store.cc:276] 0623 08:04:41.895876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.895925 (+ 49us) service_pool.cc:225] Handling call
0623 08:04:45.271032 (+3375107us) ts_tablet_manager.cc:1145] Deleting tablet 16331ba6a34942a5a1b4ff4ac7bac55e
0623 08:04:45.271044 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.277174 (+ 6130us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.286982 (+ 9808us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.329207 (+ 42225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24369,"tablet-delete.queue_time_us":3374930}
I20250623 08:04:45.330993 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 411) took 3433 ms. Trace:
I20250623 08:04:45.331213 2188 leader_election.cc:290] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.331490 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0b1287cf3b6459eafd032afc77c5824" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:45.331198 981 rpcz_store.cc:276] 0623 08:04:41.897573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.897614 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:45.330799 (+3433185us) ts_tablet_manager.cc:1145] Deleting tablet 75a6ebe0d6b0463b8ddcb1a091a2cc6f
0623 08:04:45.330808 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.330932 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3433005}
W20250623 08:04:45.331957 32589 leader_election.cc:343] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.331969 2188 raft_consensus.cc:491] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.332302 2188 raft_consensus.cc:513] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.332378 981 tablet_replica.cc:331] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.332674 343 tablet_bootstrap.cc:492] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.332875 343 ts_tablet_manager.cc:1397] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250623 08:04:45.333144 2188 leader_election.cc:290] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.333089 981 raft_consensus.cc:2241] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.333787 343 raft_consensus.cc:357] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.334038 343 raft_consensus.cc:383] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.334147 343 raft_consensus.cc:738] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.334336 2188 consensus_queue.cc:1035] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.334435 343 consensus_queue.cc:260] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.334707 981 raft_consensus.cc:2270] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.335332 2014 consensus_queue.cc:1035] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.335457 343 ts_tablet_manager.cc:1428] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:45.335790 343 tablet_bootstrap.cc:492] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:45.337246 32589 leader_election.cc:343] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: f97f14cbaffb46d681d8b11c0213cae4
I20250623 08:04:45.337328 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b720fdea539f423ba2d2aa9f89b09ae9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:45.337718 343 tablet_bootstrap.cc:654] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:45.337777 32589 leader_election.cc:343] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.333915 32314 leader_election.cc:304] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.339277 1682 raft_consensus.cc:3058] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.341898 1682 raft_consensus.cc:2747] T 6b04e38d3da44301a450fe1fb5e2ada6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.341753 981 ts_tablet_manager.cc:1905] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.344605 2242 raft_consensus.cc:491] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.344786 2242 raft_consensus.cc:513] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:45.345139 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 410) could send its response
I20250623 08:04:45.345765 2242 leader_election.cc:290] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:45.345340 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 411) could send its response
I20250623 08:04:45.346179 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea588a48e6164d3db148fdaa306268f6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.346818 32589 leader_election.cc:343] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.347265 2076 tablet_loader.cc:96] loaded metadata for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816])
I20250623 08:04:45.348297 339 tablet_bootstrap.cc:492] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.348443 339 ts_tablet_manager.cc:1397] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
I20250623 08:04:45.349066 2076 tablet_loader.cc:96] loaded metadata for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351])
I20250623 08:04:45.349263 339 raft_consensus.cc:357] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.349434 339 raft_consensus.cc:383] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.349517 339 raft_consensus.cc:738] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.349738 339 consensus_queue.cc:260] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.350217 339 ts_tablet_manager.cc:1428] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:45.350499 339 tablet_bootstrap.cc:492] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.352248 339 tablet_bootstrap.cc:654] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.350240 2076 tablet_loader.cc:96] loaded metadata for tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec])
I20250623 08:04:45.352774 2076 tablet_loader.cc:96] loaded metadata for tablet bcf753ede17f429b82612ad12e13185c (table default.Table_9968742b8af441aa8db7bd5343fb4059 [id=0e357a45fc4547d59f1f886531c3ac2c])
I20250623 08:04:45.354323 2076 tablet_loader.cc:96] loaded metadata for tablet c098dcb1c55c41ebbf6229cfe8d24a44 (table default.Table_5cfbfaf89e6a4bdf94cef7019cf9c5e4 [id=5c35ac21e1964919acdfdc5e72e5b6b7])
I20250623 08:04:45.355749 343 tablet_bootstrap.cc:492] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.355940 343 ts_tablet_manager.cc:1397] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20250623 08:04:45.356813 343 raft_consensus.cc:357] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.357029 343 raft_consensus.cc:383] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.357154 343 raft_consensus.cc:738] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.357457 343 consensus_queue.cc:260] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.358319 343 ts_tablet_manager.cc:1428] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:45.363301 343 tablet_bootstrap.cc:492] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.365250 343 tablet_bootstrap.cc:654] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.366075 2076 tablet_loader.cc:96] loaded metadata for tablet c1596e80225041099fb513ce477eae33 (table default.Table_eb7a5a786084417cb64d7093913e0e95 [id=384813cce07142a58e9c62e7ee4d54f9])
I20250623 08:04:45.366526 2076 tablet_loader.cc:96] loaded metadata for tablet c1ab3afffac646139eea609b1a22787c (table default.Table_c4c658393f424eb1ac22133856d0d0b1 [id=44217c9c758e4530aa497a405168eac0])
I20250623 08:04:45.367851 2076 tablet_loader.cc:96] loaded metadata for tablet c1b7896030ee4394bdabc785ab31b97e (table default.Table_fddbef2bc1d046689a8ef37cdcddcfa4 [id=e24b34aca192447a934b8775b709d8ac])
I20250623 08:04:45.368486 2076 tablet_loader.cc:96] loaded metadata for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_217651efa5dc4571b2a47cd3196c9e57 [id=0ea2aee143054b39ae96e30c64a42a50])
I20250623 08:04:45.371767 2076 tablet_loader.cc:96] loaded metadata for tablet c363a0602836433a8fd41fde529e96f3 (table default.Table_88d6a5158a8d4ae192cf71e8eaffff54 [id=e218dda26a86485a9983a0bc74c4c89f])
I20250623 08:04:45.372205 2076 tablet_loader.cc:96] loaded metadata for tablet c50655c06cd94c67a49513c39728eaa9 (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84])
I20250623 08:04:45.372702 2076 tablet_loader.cc:96] loaded metadata for tablet c63029399994480ba7ccabcadc1ee5e0 (table default.Table_33f948aa4f74464d86b927937ff981b9 [id=33768fa88bfc4cfba1359030af5e77c9])
W20250623 08:04:45.374080 32589 consensus_peers.cc:487] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:45.378187 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f83a6d453ec4ffe85282b4385df5f11" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.378427 32651 raft_consensus.cc:2466] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:45.379465 2076 tablet_loader.cc:96] loaded metadata for tablet c7f759b707ef46df99873b92c694ad6b (table default.Table_31ab57df769c4b2cb4e2cb74d69b4eb5 [id=92602c2df9c44292abbed7d6095922c9])
I20250623 08:04:45.381173 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23928d64bcef4c06af81698e688b7f94" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.381394 32651 raft_consensus.cc:2466] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:45.382611 2076 tablet_loader.cc:96] loaded metadata for tablet c8e4cde2af694f07a9176313b4b93a46 (table default.Table_f933517ba72b4e44bd0432107e9a9f9f [id=8a7b2a641e2f4e638cef8b6604ad12f1])
I20250623 08:04:45.383437 343 tablet_bootstrap.cc:492] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.383656 343 ts_tablet_manager.cc:1397] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250623 08:04:45.384454 343 raft_consensus.cc:357] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.384716 343 raft_consensus.cc:383] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.384835 343 raft_consensus.cc:738] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.385250 343 consensus_queue.cc:260] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.386067 343 ts_tablet_manager.cc:1428] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:45.386415 343 tablet_bootstrap.cc:492] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.387790 2076 tablet_loader.cc:96] loaded metadata for tablet ca5c694856f44201a8aa4314a4f81681 (table default.Table_b68f23253200457683bb5cedcc56041f [id=4f81957f51bd40c192ad7c716cbd893f])
I20250623 08:04:45.388422 2076 tablet_loader.cc:96] loaded metadata for tablet cac83131e6854890b803b3d89ded7cf8 (table default.Table_849f3be400444579943f3f649824f4ea [id=f223e5f800b14f08b72352e85db1a9f7])
W20250623 08:04:45.389747 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 362) could send its response
I20250623 08:04:45.389921 2076 tablet_loader.cc:96] loaded metadata for tablet cd457e5206a448d084a616cda7f410c3 (table default.Table_992eb98f4db0454083b215b3ac3d38b7 [id=743dc03385c4498c8c1ca8df7e859bee])
I20250623 08:04:45.393625 2076 tablet_loader.cc:96] loaded metadata for tablet cdc9d47f2232403fa71e14a6fe6f33c3 (table default.Table_dc973f39a76c44a1972c2b9a6c9328b8 [id=5d4bcecfd3904412b0aa6fab7c75d584])
I20250623 08:04:45.394893 2076 tablet_loader.cc:96] loaded metadata for tablet d0107b41d5c04a5c818f6792f2c18ce5 (table default.Table_91d48bbc63414bb4a1fb90f438f52eba [id=6eafeaf2ab324e3ebcf4ad4ad168cb82])
I20250623 08:04:45.396929 2076 tablet_loader.cc:96] loaded metadata for tablet d02f74c24a174b18b08a870a4e47fefb (table default.Table_a22f57b4aaa54648ab0a7c73aa2e887c [id=b34c2a31f23742838fccd1bb73863d00])
I20250623 08:04:45.398484 2076 tablet_loader.cc:96] loaded metadata for tablet d47c48aeed064015868f50c2c0dee132 (table default.Table_1802c9f70d0f45868e11e3390504fade [id=15591f6f911f4a91a1dbbef45354fe1c])
I20250623 08:04:45.399317 1957 raft_consensus.cc:491] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.399442 1957 raft_consensus.cc:513] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.399972 1957 leader_election.cc:290] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.400681 343 tablet_bootstrap.cc:654] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.402909 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7dc5e0fbeba4412aa42d116cd8882238" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.403074 32519 raft_consensus.cc:3058] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.404670 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6857437f85ab40f6a24e74ce3b5efbdc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.404834 32518 raft_consensus.cc:2466] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.404932 981 ts_tablet_manager.cc:1918] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:45.405417 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dbbdbebdf3b4b50bda31a94fc0de36f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.405548 32514 raft_consensus.cc:3058] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.405741 32519 raft_consensus.cc:2466] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.405907 981 log.cc:1199] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/b84b8b8713774ab4896842feb9fb075e
I20250623 08:04:45.407265 2076 tablet_loader.cc:96] loaded metadata for tablet d4bc93afd147458b833a026e9d0966b8 (table default.Table_f883c19d25584eb185a1e31ef6d88d4f [id=ae1d0fae75be48768a6503498a33df32])
I20250623 08:04:45.407928 2076 tablet_loader.cc:96] loaded metadata for tablet d5f82dc198af45f598bf05c05a4a8090 (table default.Table_1a1b8de8d0a645e9a48938956c562be9 [id=2562e6968e504a5aa52358337d79f528])
I20250623 08:04:45.408217 32514 raft_consensus.cc:2466] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.408481 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b61b02cf2904404f8c0ffef7d268fefc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.408646 32513 raft_consensus.cc:2466] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.408602 981 ts_tablet_manager.cc:1939] T b84b8b8713774ab4896842feb9fb075e P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:45.408849 2076 tablet_loader.cc:96] loaded metadata for tablet d6fdea59f71e4a9d9fca8ede4b9439d2 (table default.Table_b9220275c4064c11bb0618d6a459aab6 [id=392754cfbca449d887b33eb6ec5f0ae2])
I20250623 08:04:45.413383 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 412) took 3515 ms. Trace:
I20250623 08:04:45.413980 981 rpcz_store.cc:276] 0623 08:04:41.897691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.897826 (+ 135us) service_pool.cc:225] Handling call
0623 08:04:45.332295 (+3434469us) ts_tablet_manager.cc:1145] Deleting tablet b84b8b8713774ab4896842feb9fb075e
0623 08:04:45.332306 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.375502 (+ 43196us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.404130 (+ 28628us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.413186 (+ 9056us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3434264}
I20250623 08:04:45.416973 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 415) took 3518 ms. Trace:
I20250623 08:04:45.417910 2076 tablet_loader.cc:96] loaded metadata for tablet d7caf974de5049d3b6db2754970ed2e3 (table default.Table_0e43b0bc6b7b47b1b59882856aaa9736 [id=30949c1b60bc4300852bcaf533ceed4c])
I20250623 08:04:45.418421 2076 tablet_loader.cc:96] loaded metadata for tablet d98654c29b9c4e75af6a213cafd4c2c9 (table default.Table_cd6067d4c2734a3e894f926700680ca2 [id=bf814761cc7b4e739134050d1eb3c9fa])
I20250623 08:04:45.417626 981 rpcz_store.cc:276] 0623 08:04:41.897915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.897956 (+ 41us) service_pool.cc:225] Handling call
0623 08:04:45.415820 (+3517864us) ts_tablet_manager.cc:1145] Deleting tablet b61b02cf2904404f8c0ffef7d268fefc
0623 08:04:45.415990 (+ 170us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.416606 (+ 616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3517683}
I20250623 08:04:45.419559 1627 raft_consensus.cc:491] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.419713 1627 raft_consensus.cc:513] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.420301 1627 leader_election.cc:290] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.420523 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 416) took 3522 ms. Trace:
I20250623 08:04:45.421092 981 rpcz_store.cc:276] 0623 08:04:41.897993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.898039 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:45.419613 (+3521574us) ts_tablet_manager.cc:1145] Deleting tablet 89bb7ea31b4d4ce7a3c063853142fbb2
0623 08:04:45.419621 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.420201 (+ 580us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3521302}
I20250623 08:04:45.398522 342 tablet_bootstrap.cc:654] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.423568 2076 tablet_loader.cc:96] loaded metadata for tablet dcfe68d4e6ae470d816ac4c494dcaa51 (table default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe [id=f77e622d3eb44f4fa3a284b08eefc109])
I20250623 08:04:45.424285 981 tablet_replica.cc:331] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
W20250623 08:04:45.424875 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 412) could send its response
W20250623 08:04:45.425081 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 415) could send its response
W20250623 08:04:45.425427 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 416) could send its response
I20250623 08:04:45.426512 2076 tablet_loader.cc:96] loaded metadata for tablet de743139c4324faba19ff3e5cfac855d (table default.Table_d3115e0a02304f6d80807841b76504e4 [id=fc36d4cdd9684c6f84517d848ae3395b])
I20250623 08:04:45.426658 339 tablet_bootstrap.cc:492] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.426609 981 raft_consensus.cc:2241] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.427148 339 ts_tablet_manager.cc:1397] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.018s
I20250623 08:04:45.427322 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cca63991d5b4ef5bd8a9181d127ebe3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.427433 343 tablet_bootstrap.cc:492] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.427623 343 ts_tablet_manager.cc:1397] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.001s
I20250623 08:04:45.427654 1682 raft_consensus.cc:491] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.427803 1682 raft_consensus.cc:513] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.428368 1682 leader_election.cc:290] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.428445 343 raft_consensus.cc:357] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.428653 343 raft_consensus.cc:383] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.428663 981 raft_consensus.cc:2270] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.428776 343 raft_consensus.cc:738] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.429040 343 consensus_queue.cc:260] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.429608 343 ts_tablet_manager.cc:1428] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:45.429944 343 tablet_bootstrap.cc:492] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.430830 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf1f3943a3d49318c435921ebeaaf33" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.430995 32513 raft_consensus.cc:2466] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.431429 2076 tablet_loader.cc:96] loaded metadata for tablet de8130ee0d4b4603834407e52fe3f9e1 (table default.Table_fb578541a1094c0f90ba5281797a457d [id=0ed6580f5bc04a48830d33c64804a1ee])
I20250623 08:04:45.432085 2076 tablet_loader.cc:96] loaded metadata for tablet e13dc85076b34903b35310b14dc31085 (table default.Table_315a89d642a74362b8a04f8b82edd22f [id=7f7d25e8c0904f09b3cbd93034adcb0d])
I20250623 08:04:45.432638 2076 tablet_loader.cc:96] loaded metadata for tablet e259ef662ee44d1ca5081c05e9fed5f6 (table default.Table_bc37bf6911b5401c904eef106dde3592 [id=19ae0916b8c6480f93b92da98de7ec4d])
I20250623 08:04:45.433025 2076 tablet_loader.cc:96] loaded metadata for tablet e27da2a90c1b43deb3dc90834f4e312e (table default.Table_de7a3df7f0064a7bb7715c0c5dcc337e [id=5e7083acd09b41b2b78bd229dc7b1e3d])
I20250623 08:04:45.433389 2076 tablet_loader.cc:96] loaded metadata for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d])
I20250623 08:04:45.433800 2076 tablet_loader.cc:96] loaded metadata for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc])
I20250623 08:04:45.434597 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0b1287cf3b6459eafd032afc77c5824" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.434775 32513 raft_consensus.cc:3058] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.434268 339 raft_consensus.cc:357] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.435223 339 raft_consensus.cc:383] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.435582 343 tablet_bootstrap.cc:654] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.435636 339 raft_consensus.cc:738] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.437475 32513 raft_consensus.cc:2466] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.438104 981 ts_tablet_manager.cc:1905] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.438690 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b720fdea539f423ba2d2aa9f89b09ae9" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.437155 339 consensus_queue.cc:260] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.438333 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f83a6d453ec4ffe85282b4385df5f11" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:45.439221 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23928d64bcef4c06af81698e688b7f94" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:45.444336 339 ts_tablet_manager.cc:1428] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250623 08:04:45.447013 32378 raft_consensus.cc:3058] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.447108 339 tablet_bootstrap.cc:492] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.447481 342 tablet_bootstrap.cc:492] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:45.447609 342 ts_tablet_manager.cc:1397] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.131s user 0.003s sys 0.000s
I20250623 08:04:45.448206 342 raft_consensus.cc:357] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.448356 342 raft_consensus.cc:383] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.448400 342 raft_consensus.cc:738] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.448592 342 consensus_queue.cc:260] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.449066 342 ts_tablet_manager.cc:1428] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:45.449846 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea588a48e6164d3db148fdaa306268f6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.444028 2076 tablet_loader.cc:96] loaded metadata for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07])
I20250623 08:04:45.449987 32514 raft_consensus.cc:2466] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.450804 2076 tablet_loader.cc:96] loaded metadata for tablet e9604d4890c940d8992eaf30993bf622 (table default.Table_dd2b39480694444ba170c65257a8dc58 [id=9dd60e056f8b4c17b9d8d9569ebd3447])
I20250623 08:04:45.450834 2014 consensus_queue.cc:1035] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.451175 2076 tablet_loader.cc:96] loaded metadata for tablet e9a0c57133884896b03e9c0a8afb8bde (table default.Table_63b537810a3a4f478e4226cabee7004d [id=692d14d5e7224e37b0a78ac3171bfefa])
I20250623 08:04:45.451378 32378 raft_consensus.cc:3058] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.451977 2076 tablet_loader.cc:96] loaded metadata for tablet ea588a48e6164d3db148fdaa306268f6 (table default.Table_b523bec7182443139e891357d295e3a1 [id=1c4298a5f9014d8eb4ccad8c8fbdeeac])
I20250623 08:04:45.452576 2076 tablet_loader.cc:96] loaded metadata for tablet ea77b97018b74708bd6057bd2d65f90b (table default.Table_5338d405e4ed40589c691093043355d1 [id=b9246ed2ad6242739ae14a362b4af828])
I20250623 08:04:45.452891 2076 tablet_loader.cc:96] loaded metadata for tablet ebe6148c11144fdb8519a3973e822159 (table default.Table_306d24f2f3484139b3586e0a1ebb8562 [id=2f72a0d782b54f109937d1ab340afd89])
I20250623 08:04:45.453243 2076 tablet_loader.cc:96] loaded metadata for tablet ec4d50d5f9394afe97e3b43d65e6894e (table default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5])
I20250623 08:04:45.453796 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9891997e026448e39943ae13082bfe03" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.453851 2076 tablet_loader.cc:96] loaded metadata for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9])
I20250623 08:04:45.454000 32519 raft_consensus.cc:2391] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:45.454631 2076 tablet_loader.cc:96] loaded metadata for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b])
I20250623 08:04:45.455981 342 tablet_bootstrap.cc:492] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:45.456712 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9891997e026448e39943ae13082bfe03" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.457912 342 tablet_bootstrap.cc:654] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.458180 339 tablet_bootstrap.cc:654] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.460862 32314 leader_election.cc:304] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.464270 2229 raft_consensus.cc:3058] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.467701 2229 raft_consensus.cc:2747] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.469381 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf1f3943a3d49318c435921ebeaaf33" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.469724 32645 raft_consensus.cc:2466] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:45.469833 342 tablet_bootstrap.cc:492] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:45.471673 2014 raft_consensus.cc:491] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.471807 2014 raft_consensus.cc:513] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.472375 2014 leader_election.cc:290] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.473481 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "077bb090b3ea4d029463304d7a6feee5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.474102 32589 leader_election.cc:343] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.474946 342 ts_tablet_manager.cc:1397] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20250623 08:04:45.475762 342 raft_consensus.cc:357] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.476400 2076 tablet_loader.cc:96] loaded metadata for tablet eef663ffdaf141aea5e6269b6b2e1c89 (table default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d])
I20250623 08:04:45.476625 343 tablet_bootstrap.cc:492] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.476811 343 ts_tablet_manager.cc:1397] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.003s
I20250623 08:04:45.477500 343 raft_consensus.cc:357] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.477741 343 raft_consensus.cc:383] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.477882 343 raft_consensus.cc:738] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.478243 343 consensus_queue.cc:260] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.479588 343 ts_tablet_manager.cc:1428] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:45.480530 342 raft_consensus.cc:383] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.480651 342 raft_consensus.cc:738] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.480926 343 tablet_bootstrap.cc:492] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.480888 342 consensus_queue.cc:260] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.481950 2076 tablet_loader.cc:96] loaded metadata for tablet f080a682e532406a83712b6f3e6d2f7b (table default.Table_fea0e0f4beec4bddb07a0d24efe95aa4 [id=d869eced4268436fbf4ba0666f3b7790])
I20250623 08:04:45.482549 342 ts_tablet_manager.cc:1428] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250623 08:04:45.482880 342 tablet_bootstrap.cc:492] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:45.483002 32589 leader_election.cc:304] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.483285 2014 raft_consensus.cc:2802] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.483353 2014 raft_consensus.cc:491] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.483414 2014 raft_consensus.cc:3058] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.483739 32589 leader_election.cc:304] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.483800 2076 tablet_loader.cc:96] loaded metadata for tablet f1113750ec7a4dfe8a424afe0d41c33b (table default.Table_eb0f697365cc4f2b8bbeb5b3de47f1dc [id=f15f4f7ed5054a3ca292217b18a14a94])
I20250623 08:04:45.483958 2242 raft_consensus.cc:2802] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:45.484066 2242 raft_consensus.cc:695] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:45.484277 2242 consensus_queue.cc:237] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.484568 32589 leader_election.cc:304] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.484838 2210 raft_consensus.cc:2802] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:45.484930 2210 raft_consensus.cc:695] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:45.485037 32589 leader_election.cc:304] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.485134 2210 consensus_queue.cc:237] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.485323 2188 raft_consensus.cc:2802] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.485392 2188 raft_consensus.cc:491] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.485461 2188 raft_consensus.cc:3058] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.486148 2229 raft_consensus.cc:491] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.486291 2229 raft_consensus.cc:513] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.486883 2229 leader_election.cc:290] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.488343 2188 raft_consensus.cc:513] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.489645 2188 leader_election.cc:290] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.487982 2014 raft_consensus.cc:513] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.490675 2014 leader_election.cc:290] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.492272 343 tablet_bootstrap.cc:654] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.497663 2076 tablet_loader.cc:96] loaded metadata for tablet f20b23fd6e5c44f58b1fb2bf617c5704 (table default.Table_744bf9c83e004aafb60d2cb44530dff5 [id=dd185047529940ebbd12e27a9f6577a6])
I20250623 08:04:45.501029 952 ts_tablet_manager.cc:1918] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:45.501243 952 log.cc:1199] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/90ceb0f8d5c94c4e9427042d8a35e16c
I20250623 08:04:45.503326 2076 tablet_loader.cc:96] loaded metadata for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163])
I20250623 08:04:45.505926 342 tablet_bootstrap.cc:654] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.508976 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf1f3943a3d49318c435921ebeaaf33" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:45.510217 32368 raft_consensus.cc:3058] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.513372 343 tablet_bootstrap.cc:492] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.513504 2076 tablet_loader.cc:96] loaded metadata for tablet f44fd3c67185484e93450682364dc8a8 (table default.Table_e2502c251f3c4a7f8caa4faab352ebda [id=664b85653cd341a6a6afe7d04fb9c2ac])
I20250623 08:04:45.513712 343 ts_tablet_manager.cc:1397] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20250623 08:04:45.514952 2076 tablet_loader.cc:96] loaded metadata for tablet f7d69bcad8684d99a0b08eefb470ab01 (table default.Table_0bb2584ec3e0467194a9591bbe5ae477 [id=87e7893d85264d4c9cbcc5564f840390])
I20250623 08:04:45.515619 2076 tablet_loader.cc:96] loaded metadata for tablet f8a5e35c1974456b8079ab8907c2652e (table default.Table_d2473e1e63c84b229243a1c676266ca6 [id=c63ada5e02d74e049460990450ab4962])
I20250623 08:04:45.515625 343 raft_consensus.cc:357] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.516108 343 raft_consensus.cc:383] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.516309 343 raft_consensus.cc:738] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.516532 2076 tablet_loader.cc:96] loaded metadata for tablet fa41bab84b384c10ace3ce176698ebe4 (table default.Table_cace1e64be664c9c81b588e80d704aaa [id=7ed25123fbc749909d1d90134be64278])
I20250623 08:04:45.516860 343 consensus_queue.cc:260] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.517632 342 tablet_bootstrap.cc:492] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:45.518033 343 ts_tablet_manager.cc:1428] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250623 08:04:45.518715 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b61b02cf2904404f8c0ffef7d268fefc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:45.519380 342 ts_tablet_manager.cc:1397] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20250623 08:04:45.519440 343 tablet_bootstrap.cc:492] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.519912 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6857437f85ab40f6a24e74ce3b5efbdc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:45.520834 342 raft_consensus.cc:357] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:45.522081 32589 leader_election.cc:343] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.522325 32589 leader_election.cc:343] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.522481 342 raft_consensus.cc:383] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.523048 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "077bb090b3ea4d029463304d7a6feee5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.523144 342 raft_consensus.cc:738] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.523223 32651 raft_consensus.cc:2466] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:45.523501 342 consensus_queue.cc:260] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.525178 342 ts_tablet_manager.cc:1428] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
W20250623 08:04:45.525326 32589 leader_election.cc:343] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.525454 32589 leader_election.cc:304] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.525825 342 tablet_bootstrap.cc:492] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:45.526648 32589 leader_election.cc:304] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.526927 2329 raft_consensus.cc:2747] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.528066 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "077bb090b3ea4d029463304d7a6feee5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:45.528820 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dbbdbebdf3b4b50bda31a94fc0de36f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.528889 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dbbdbebdf3b4b50bda31a94fc0de36f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.529035 32513 raft_consensus.cc:2391] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:45.529523 32314 leader_election.cc:304] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.529665 343 tablet_bootstrap.cc:654] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.530509 32589 leader_election.cc:304] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:45.531810 32589 leader_election.cc:343] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.532059 32589 leader_election.cc:304] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:45.532670 342 tablet_bootstrap.cc:654] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.533038 32589 leader_election.cc:304] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:45.533601 32589 consensus_peers.cc:487] T 90ceb0f8d5c94c4e9427042d8a35e16c P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:45.531985 2329 raft_consensus.cc:2802] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:45.541461 2338 raft_consensus.cc:2802] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.541666 2014 consensus_queue.cc:1035] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.541743 2338 raft_consensus.cc:491] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.541966 2338 raft_consensus.cc:3058] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.542296 2242 raft_consensus.cc:2747] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.548446 1682 raft_consensus.cc:491] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:45.548624 1682 raft_consensus.cc:513] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.549409 1682 leader_election.cc:290] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.551963 32453 leader_election.cc:304] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:45.552240 1957 raft_consensus.cc:2802] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.552309 1957 raft_consensus.cc:491] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.552366 1957 raft_consensus.cc:3058] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.555083 1957 raft_consensus.cc:513] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.555706 1957 leader_election.cc:290] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.557017 2338 raft_consensus.cc:513] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.559201 2076 tablet_loader.cc:96] loaded metadata for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa])
W20250623 08:04:45.559891 2076 catalog_manager.cc:1478] Time spent T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Loading table and tablet metadata into memory: real 2.456s user 0.253s sys 0.071s
I20250623 08:04:45.562491 952 ts_tablet_manager.cc:1939] T 90ceb0f8d5c94c4e9427042d8a35e16c P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:45.560338 1248 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:57678 (request call id 0) took 1835 ms. Trace:
I20250623 08:04:45.563215 1248 rpcz_store.cc:276] 0623 08:04:43.724675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:43.724744 (+ 69us) service_pool.cc:225] Handling call
0623 08:04:45.560273 (+1835529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 08:04:45.563403 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 364) took 5880 ms. Trace:
I20250623 08:04:45.563570 952 rpcz_store.cc:276] 0623 08:04:39.683097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.683282 (+ 185us) service_pool.cc:225] Handling call
0623 08:04:45.303480 (+5620198us) ts_tablet_manager.cc:1145] Deleting tablet 90ceb0f8d5c94c4e9427042d8a35e16c
0623 08:04:45.303494 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.413270 (+109776us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.500888 (+ 87618us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.563383 (+ 62495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5619994}
I20250623 08:04:45.563954 2338 leader_election.cc:290] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.563966 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 368) took 5766 ms. Trace:
I20250623 08:04:45.564081 952 rpcz_store.cc:276] 0623 08:04:39.797060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.797114 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:45.563853 (+5766739us) ts_tablet_manager.cc:1145] Deleting tablet aface2a39d764a6cb6e4965867ab7b38
0623 08:04:45.563862 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.563954 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5760144}
I20250623 08:04:45.564220 2076 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250623 08:04:45.564460 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 370) took 5766 ms. Trace:
I20250623 08:04:45.564576 952 rpcz_store.cc:276] 0623 08:04:39.798066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.798398 (+ 332us) service_pool.cc:225] Handling call
0623 08:04:45.564338 (+5765940us) ts_tablet_manager.cc:1145] Deleting tablet 1aa604f500984e3b969e15592cdbed6f
0623 08:04:45.564349 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.564444 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5734338}
I20250623 08:04:45.564949 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 372) took 5640 ms. Trace:
I20250623 08:04:45.566004 981 ts_tablet_manager.cc:1918] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:45.566185 981 log.cc:1199] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/36c8714cfbd84dd38cf5c0f305ef6843
I20250623 08:04:45.568801 339 tablet_bootstrap.cc:492] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.570030 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90ceb0f8d5c94c4e9427042d8a35e16c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.570971 339 ts_tablet_manager.cc:1397] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.124s user 0.006s sys 0.014s
I20250623 08:04:45.569082 2229 raft_consensus.cc:3058] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.572980 339 raft_consensus.cc:357] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.576262 339 raft_consensus.cc:383] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.576808 339 raft_consensus.cc:738] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.577610 2229 raft_consensus.cc:2747] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.578136 339 consensus_queue.cc:260] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.579568 339 ts_tablet_manager.cc:1428] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250623 08:04:45.581200 339 tablet_bootstrap.cc:492] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:45.584879 32314 leader_election.cc:343] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 90ceb0f8d5c94c4e9427042d8a35e16c
I20250623 08:04:45.585191 32314 leader_election.cc:304] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.592165 343 tablet_bootstrap.cc:492] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.592370 343 ts_tablet_manager.cc:1397] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.073s user 0.013s sys 0.001s
I20250623 08:04:45.593127 343 raft_consensus.cc:357] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.593369 343 raft_consensus.cc:383] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.593472 343 raft_consensus.cc:738] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.593724 343 consensus_queue.cc:260] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.594316 343 ts_tablet_manager.cc:1428] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:45.594614 343 tablet_bootstrap.cc:492] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.595963 343 tablet_bootstrap.cc:654] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.596613 2329 raft_consensus.cc:695] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:45.597276 2329 consensus_queue.cc:237] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:45.597530 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 58906.8us
I20250623 08:04:45.616992 2329 consensus_queue.cc:1035] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.617362 2329 raft_consensus.cc:491] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.617466 2329 raft_consensus.cc:513] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.618002 2329 leader_election.cc:290] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.620747 952 rpcz_store.cc:276] 0623 08:04:39.924353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.924770 (+ 417us) service_pool.cc:225] Handling call
0623 08:04:45.564847 (+5640077us) ts_tablet_manager.cc:1145] Deleting tablet 185c2bf7b9254947bb76b7fe0924559e
0623 08:04:45.564857 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.564934 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5639807}
I20250623 08:04:45.621150 952 tablet_replica.cc:331] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:45.621487 952 raft_consensus.cc:2241] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.621687 952 raft_consensus.cc:2270] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.568843 32453 leader_election.cc:304] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
W20250623 08:04:45.622661 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 53841.8us
I20250623 08:04:45.622707 1957 raft_consensus.cc:2802] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.622853 1957 raft_consensus.cc:491] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.623238 1957 raft_consensus.cc:3058] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.624415 343 tablet_bootstrap.cc:492] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.624609 343 ts_tablet_manager.cc:1397] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250623 08:04:45.625288 343 raft_consensus.cc:357] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.625473 343 raft_consensus.cc:383] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.625586 343 raft_consensus.cc:738] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.625823 343 consensus_queue.cc:260] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.626070 32453 leader_election.cc:304] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:45.626574 1823 raft_consensus.cc:2802] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.626662 1823 raft_consensus.cc:491] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.626780 1823 raft_consensus.cc:3058] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.627048 343 ts_tablet_manager.cc:1428] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:45.627353 343 tablet_bootstrap.cc:492] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.628880 1957 raft_consensus.cc:513] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.629580 1823 raft_consensus.cc:513] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.630817 1823 leader_election.cc:290] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.631187 343 tablet_bootstrap.cc:654] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.631608 1957 leader_election.cc:290] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.587466 339 tablet_bootstrap.cc:654] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:45.635241 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 50061.9us
I20250623 08:04:45.635962 2260 raft_consensus.cc:2747] T 90ceb0f8d5c94c4e9427042d8a35e16c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:45.597309 981 ts_tablet_manager.cc:1939] T 36c8714cfbd84dd38cf5c0f305ef6843 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:45.599351 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea588a48e6164d3db148fdaa306268f6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:45.620983 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baccec88aefb4056be9578bca13f1591" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.643952 32589 leader_election.cc:343] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.644189 32589 leader_election.cc:343] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.647909 2336 consensus_queue.cc:1035] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.649304 343 tablet_bootstrap.cc:492] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.649485 343 ts_tablet_manager.cc:1397] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.004s
I20250623 08:04:45.650257 343 raft_consensus.cc:357] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.650488 343 raft_consensus.cc:383] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.650596 343 raft_consensus.cc:738] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.650856 343 consensus_queue.cc:260] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.651605 343 ts_tablet_manager.cc:1428] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:45.652050 343 tablet_bootstrap.cc:492] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.653604 343 tablet_bootstrap.cc:654] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.660643 2076 catalog_manager.cc:1261] Loaded cluster ID: bde910f56f35432e94ef19d006edb93c
I20250623 08:04:45.663786 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 413) took 3765 ms. Trace:
I20250623 08:04:45.664007 981 rpcz_store.cc:276] 0623 08:04:41.897763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.897973 (+ 210us) service_pool.cc:225] Handling call
0623 08:04:45.423827 (+3525854us) ts_tablet_manager.cc:1145] Deleting tablet 36c8714cfbd84dd38cf5c0f305ef6843
0623 08:04:45.423999 (+ 172us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.480128 (+ 56129us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.565859 (+ 85731us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.663699 (+ 97840us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1196,"tablet-delete.queue_time_us":3525344}
I20250623 08:04:45.664510 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 417) took 3766 ms. Trace:
W20250623 08:04:45.664772 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 413) could send its response
I20250623 08:04:45.664620 981 rpcz_store.cc:276] 0623 08:04:41.898089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.898148 (+ 59us) service_pool.cc:225] Handling call
0623 08:04:45.664345 (+3766197us) ts_tablet_manager.cc:1145] Deleting tablet 2f62ec83c88e43438b963e740d2c7004
0623 08:04:45.664356 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.664466 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3766061}
I20250623 08:04:45.666392 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 418) took 3768 ms. Trace:
I20250623 08:04:45.666478 981 rpcz_store.cc:276] 0623 08:04:41.898232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.898263 (+ 31us) service_pool.cc:225] Handling call
0623 08:04:45.666285 (+3768022us) ts_tablet_manager.cc:1145] Deleting tablet 18ea63263b9b466e922169eecafab747
0623 08:04:45.666295 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.666360 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3767917}
I20250623 08:04:45.666622 342 tablet_bootstrap.cc:492] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:45.666774 342 ts_tablet_manager.cc:1397] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.022s
W20250623 08:04:45.668736 32453 leader_election.cc:343] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.669080 32453 leader_election.cc:343] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.666728 981 tablet_replica.cc:331] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.679538 981 raft_consensus.cc:2241] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.680507 981 raft_consensus.cc:2270] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.681068 981 ts_tablet_manager.cc:1905] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.680359 339 tablet_bootstrap.cc:492] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.685007 339 ts_tablet_manager.cc:1397] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.104s user 0.001s sys 0.013s
I20250623 08:04:45.685562 2336 raft_consensus.cc:491] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.685748 2336 raft_consensus.cc:513] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.685957 952 ts_tablet_manager.cc:1905] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.687400 2336 leader_election.cc:290] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.687582 2336 raft_consensus.cc:491] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.687731 2336 raft_consensus.cc:513] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.688997 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8911308855cb4414be426dbcec592f4a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.689533 32589 leader_election.cc:343] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.689906 2076 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250623 08:04:45.690007 2336 leader_election.cc:290] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.690279 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f5f58f55070344ff98a3808f70e0c755" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.690565 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 417) could send its response
W20250623 08:04:45.690770 32589 leader_election.cc:343] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.691092 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 418) could send its response
I20250623 08:04:45.694047 342 raft_consensus.cc:357] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.696838 2336 raft_consensus.cc:491] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.697054 2336 raft_consensus.cc:513] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.696854 343 tablet_bootstrap.cc:492] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.697927 343 ts_tablet_manager.cc:1397] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.004s
I20250623 08:04:45.698191 2348 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9752155c20ee42fbb7ff7b7d5b682444)
I20250623 08:04:45.698385 2348 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
W20250623 08:04:45.698635 32589 leader_election.cc:343] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 2c5ecd4fa5574a1b90b6e82f89ee88f7
I20250623 08:04:45.699119 2336 leader_election.cc:290] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.699573 343 raft_consensus.cc:357] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.700500 343 raft_consensus.cc:383] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.700685 343 raft_consensus.cc:738] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.701081 343 consensus_queue.cc:260] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.704157 343 ts_tablet_manager.cc:1428] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:45.704808 343 tablet_bootstrap.cc:492] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.708534 343 tablet_bootstrap.cc:654] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.691499 339 raft_consensus.cc:357] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.708942 339 raft_consensus.cc:383] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.709093 339 raft_consensus.cc:738] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.710279 339 consensus_queue.cc:260] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.711467 339 ts_tablet_manager.cc:1428] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20250623 08:04:45.711848 339 tablet_bootstrap.cc:492] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.713969 339 tablet_bootstrap.cc:654] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.715245 342 raft_consensus.cc:383] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.717772 981 ts_tablet_manager.cc:1918] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:45.719131 2229 raft_consensus.cc:491] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.719285 2229 raft_consensus.cc:513] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.719390 342 raft_consensus.cc:738] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.719648 342 consensus_queue.cc:260] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:45.712821 32453 consensus_peers.cc:487] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b11777e0bd04e61a5bc8bb6ec4d97c3. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:45.719853 2229 leader_election.cc:290] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.720965 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89bb7ea31b4d4ce7a3c063853142fbb2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.723445 1267 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 5 candidate_status { last_received { term: 4 index: 761 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
I20250623 08:04:45.725019 981 log.cc:1199] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/501b5233522f43c7aed3c9770adb5c03
I20250623 08:04:45.725467 981 ts_tablet_manager.cc:1939] T 501b5233522f43c7aed3c9770adb5c03 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:45.731591 32453 leader_election.cc:343] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.733824 342 ts_tablet_manager.cc:1428] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20250623 08:04:45.734141 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 419) took 3833 ms. Trace:
I20250623 08:04:45.734304 981 rpcz_store.cc:276] 0623 08:04:41.898325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.898361 (+ 36us) service_pool.cc:225] Handling call
0623 08:04:45.666655 (+3768294us) ts_tablet_manager.cc:1145] Deleting tablet 501b5233522f43c7aed3c9770adb5c03
0623 08:04:45.666664 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.684713 (+ 18049us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.717635 (+ 32922us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.731526 (+ 13891us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3768172}
I20250623 08:04:45.734864 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 414) took 3836 ms. Trace:
I20250623 08:04:45.734997 981 rpcz_store.cc:276] 0623 08:04:41.897843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.898399 (+ 556us) service_pool.cc:225] Handling call
0623 08:04:45.734673 (+3836274us) ts_tablet_manager.cc:1145] Deleting tablet b81263a9ee9d4da1ac721c3b2bd21248
0623 08:04:45.734694 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.734809 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3836155}
W20250623 08:04:45.735802 385 leader_election.cc:336] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d31835ec4ad74332a00998d431492466 (127.31.69.62:39465): Network error: Client connection negotiation failed: client connection to 127.31.69.62:39465: connect: Connection refused (error 111)
I20250623 08:04:45.736003 385 leader_election.cc:304] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678; no voters: 9752155c20ee42fbb7ff7b7d5b682444, d31835ec4ad74332a00998d431492466
I20250623 08:04:45.737314 2348 leader_election.cc:290] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313)
I20250623 08:04:45.737596 2348 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250623 08:04:45.710902 32589 consensus_peers.cc:487] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:45.742336 2197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250623 08:04:45.742789 2197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250623 08:04:45.742928 2197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250623 08:04:45.713413 2336 raft_consensus.cc:491] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.738370 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89bb7ea31b4d4ce7a3c063853142fbb2" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.745997 2336 raft_consensus.cc:513] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.746682 2336 leader_election.cc:290] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.751030 339 tablet_bootstrap.cc:492] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.751202 339 ts_tablet_manager.cc:1397] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.002s
I20250623 08:04:45.751411 342 tablet_bootstrap.cc:492] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:45.752023 339 raft_consensus.cc:357] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.750957 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7b9e843ff1344a1aaacf038f15f64cd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:45.752231 339 raft_consensus.cc:383] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.752365 339 raft_consensus.cc:738] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:45.737150 32453 leader_election.cc:343] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.752668 32589 leader_election.cc:343] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.753938 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 419) could send its response
I20250623 08:04:45.754973 32381 raft_consensus.cc:3058] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.756304 343 tablet_bootstrap.cc:492] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.756474 343 ts_tablet_manager.cc:1397] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20250623 08:04:45.757243 343 raft_consensus.cc:357] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.757431 343 raft_consensus.cc:383] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.757517 343 raft_consensus.cc:738] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.757778 343 consensus_queue.cc:260] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.758285 343 ts_tablet_manager.cc:1428] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:45.758553 343 tablet_bootstrap.cc:492] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.759994 32314 leader_election.cc:304] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.761354 343 tablet_bootstrap.cc:654] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.762957 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f83a6d453ec4ffe85282b4385df5f11" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:45.765475 32654 raft_consensus.cc:3058] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.766922 2076 catalog_manager.cc:1506] Loading token signing keys...
I20250623 08:04:45.768349 32654 raft_consensus.cc:2466] T 8f83a6d453ec4ffe85282b4385df5f11 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:45.769593 1957 raft_consensus.cc:491] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.769742 1957 raft_consensus.cc:513] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.770375 1957 leader_election.cc:290] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.772415 342 tablet_bootstrap.cc:654] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.752895 339 consensus_queue.cc:260] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:45.774768 32589 consensus_peers.cc:487] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3136ec837bea4619ae3995ceec9b24d6. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:45.775475 339 ts_tablet_manager.cc:1428] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20250623 08:04:45.776005 339 tablet_bootstrap.cc:492] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.777594 339 tablet_bootstrap.cc:654] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.777578 343 tablet_bootstrap.cc:492] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.777758 343 ts_tablet_manager.cc:1397] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250623 08:04:45.779170 2229 raft_consensus.cc:2747] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.779711 339 tablet_bootstrap.cc:492] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.779889 339 ts_tablet_manager.cc:1397] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
W20250623 08:04:45.780211 385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 44418.6us
I20250623 08:04:45.780696 339 raft_consensus.cc:357] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.781224 339 raft_consensus.cc:383] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.781427 339 raft_consensus.cc:738] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.782140 339 consensus_queue.cc:260] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.783228 339 ts_tablet_manager.cc:1428] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:45.783324 343 raft_consensus.cc:357] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.783491 343 raft_consensus.cc:383] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.783641 343 raft_consensus.cc:738] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.783847 343 consensus_queue.cc:260] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.783985 339 tablet_bootstrap.cc:492] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.785054 2210 raft_consensus.cc:2802] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:45.785300 2210 raft_consensus.cc:491] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:45.785374 2210 raft_consensus.cc:3058] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.787921 2210 raft_consensus.cc:513] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.788506 2210 leader_election.cc:290] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.785050 343 ts_tablet_manager.cc:1428] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:45.789037 343 tablet_bootstrap.cc:492] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.789037 2076 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Loaded TSK: 0
I20250623 08:04:45.790436 343 tablet_bootstrap.cc:654] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.793551 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf1f3943a3d49318c435921ebeaaf33" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:45.794071 32589 leader_election.cc:343] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.797202 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f83a6d453ec4ffe85282b4385df5f11" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:45.800771 339 tablet_bootstrap.cc:654] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.801949 2076 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250623 08:04:45.812063 1213 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444 -> Peer d31835ec4ad74332a00998d431492466 (127.31.69.62:39465): Couldn't send request to peer d31835ec4ad74332a00998d431492466. Status: Network error: Client connection negotiation failed: client connection to 127.31.69.62:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250623 08:04:45.814234 2014 raft_consensus.cc:491] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.814386 2014 raft_consensus.cc:513] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.814929 2014 leader_election.cc:290] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.815852 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cd316e6f4447be958611ab07ed31fc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.816427 32589 leader_election.cc:343] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.816859 339 tablet_bootstrap.cc:492] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:45.817010 339 ts_tablet_manager.cc:1397] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250623 08:04:45.817772 339 raft_consensus.cc:357] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.821662 343 tablet_bootstrap.cc:492] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.821817 343 ts_tablet_manager.cc:1397] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20250623 08:04:45.822588 343 raft_consensus.cc:357] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.822793 343 raft_consensus.cc:383] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.822856 343 raft_consensus.cc:738] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.823074 343 consensus_queue.cc:260] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.824239 343 ts_tablet_manager.cc:1428] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:45.824533 343 tablet_bootstrap.cc:492] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:45.778903 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 364) could send its response
W20250623 08:04:45.828701 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 368) could send its response
W20250623 08:04:45.828863 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 370) could send its response
I20250623 08:04:45.831629 2229 raft_consensus.cc:491] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250623 08:04:45.831812 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 372) could send its response
I20250623 08:04:45.831871 2229 raft_consensus.cc:513] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.832438 339 raft_consensus.cc:383] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.832531 339 raft_consensus.cc:738] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.832731 339 consensus_queue.cc:260] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.833109 343 tablet_bootstrap.cc:654] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.834856 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "077bb090b3ea4d029463304d7a6feee5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:45.835017 32654 raft_consensus.cc:3058] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.836781 339 ts_tablet_manager.cc:1428] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250623 08:04:45.836956 2229 leader_election.cc:290] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.837553 32654 raft_consensus.cc:2466] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:45.839118 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 420) took 3940 ms. Trace:
I20250623 08:04:45.839502 2014 raft_consensus.cc:491] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.839499 981 rpcz_store.cc:276] 0623 08:04:41.898430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.898537 (+ 107us) service_pool.cc:225] Handling call
0623 08:04:45.836731 (+3938194us) ts_tablet_manager.cc:1145] Deleting tablet f5f58f55070344ff98a3808f70e0c755
0623 08:04:45.838987 (+ 2256us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.839102 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3938041}
I20250623 08:04:45.839622 2014 raft_consensus.cc:513] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.839833 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 421) took 3941 ms. Trace:
I20250623 08:04:45.839902 981 rpcz_store.cc:276] 0623 08:04:41.898944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.899058 (+ 114us) service_pool.cc:225] Handling call
0623 08:04:45.839735 (+3940677us) ts_tablet_manager.cc:1145] Deleting tablet b720fdea539f423ba2d2aa9f89b09ae9
0623 08:04:45.839744 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.839817 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3940521}
I20250623 08:04:45.840130 2014 leader_election.cc:290] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.840157 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 409) took 3944 ms. Trace:
I20250623 08:04:45.840222 981 rpcz_store.cc:276] 0623 08:04:41.895761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.905150 (+ 9389us) service_pool.cc:225] Handling call
0623 08:04:45.840081 (+3934931us) ts_tablet_manager.cc:1145] Deleting tablet 26f4a1b472e848ccb9a1ec71c1dd3516
0623 08:04:45.840091 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.840144 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3934747}
I20250623 08:04:45.840582 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 422) took 3850 ms. Trace:
I20250623 08:04:45.840678 981 rpcz_store.cc:276] 0623 08:04:41.989667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990186 (+ 519us) service_pool.cc:225] Handling call
0623 08:04:45.840489 (+3850303us) ts_tablet_manager.cc:1145] Deleting tablet ba5913811e9343389c9ea0ec8b8c05ef
0623 08:04:45.840497 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.840569 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3750566}
I20250623 08:04:45.840978 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 427) took 3850 ms. Trace:
I20250623 08:04:45.841107 981 rpcz_store.cc:276] 0623 08:04:41.990153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990197 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:45.840904 (+3850707us) ts_tablet_manager.cc:1145] Deleting tablet 48da1e4759bb4fea87a45ad8a428b6e7
0623 08:04:45.840912 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.840965 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3750875}
I20250623 08:04:45.841223 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "185c2bf7b9254947bb76b7fe0924559e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:45.841391 981 tablet_replica.cc:331] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.841624 981 raft_consensus.cc:2241] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.841826 981 raft_consensus.cc:2270] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:45.841871 32589 leader_election.cc:343] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:45.842402 981 ts_tablet_manager.cc:1905] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.842550 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23928d64bcef4c06af81698e688b7f94" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:45.842775 32654 raft_consensus.cc:3058] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.845404 32654 raft_consensus.cc:2466] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:45.839440 339 tablet_bootstrap.cc:492] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:45.852083 339 tablet_bootstrap.cc:654] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.852803 343 tablet_bootstrap.cc:492] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.852957 343 ts_tablet_manager.cc:1397] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20250623 08:04:45.853654 343 raft_consensus.cc:357] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.853813 343 raft_consensus.cc:383] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.853863 343 raft_consensus.cc:738] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.854050 343 consensus_queue.cc:260] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.858933 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.859205 32516 raft_consensus.cc:2391] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:45.861372 1242 master_service.cc:948] ReplaceTablet: received request to replace tablet dcfe68d4e6ae470d816ac4c494dcaa51 from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:45.861903 1235 master_service.cc:948] ReplaceTablet: received request to replace tablet 30dfa986dfd44658a0ffb07796030520 from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:45.862244 1245 master_service.cc:948] ReplaceTablet: received request to replace tablet 9fd478c007914d2cb09d929b50322a64 from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:45.863189 1231 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_d6a3e0399f3b420c821b3393e20499df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:45.865732 1231 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6a3e0399f3b420c821b3393e20499df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:45.867097 1230 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_2efaa95bf059477fb226d82901892c43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:45.867950 1230 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2efaa95bf059477fb226d82901892c43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:45.868944 342 tablet_bootstrap.cc:492] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:45.869061 342 ts_tablet_manager.cc:1397] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.118s user 0.001s sys 0.002s
I20250623 08:04:45.869747 342 raft_consensus.cc:357] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.869889 342 raft_consensus.cc:383] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.869928 342 raft_consensus.cc:738] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.869938 1229 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_905d0330c27149a19dabb586c51a2e0a" } new_table_name: "default.Table_53879d6f821f4167bf02ffbe967f73f7" modify_external_catalogs: true
I20250623 08:04:45.870074 342 consensus_queue.cc:260] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.871290 952 ts_tablet_manager.cc:1918] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:45.871397 952 log.cc:1199] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9dbbdbebdf3b4b50bda31a94fc0de36f
I20250623 08:04:45.871735 952 ts_tablet_manager.cc:1939] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:45.872303 1227 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_b523bec7182443139e891357d295e3a1" } modify_external_catalogs: true
I20250623 08:04:45.872537 1227 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_b523bec7182443139e891357d295e3a1" } modify_external_catalogs: true
I20250623 08:04:45.877215 1225 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_e0696987af214f9fb46f9c76b529fca5" } modify_external_catalogs: true
I20250623 08:04:45.877276 1226 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_bbc25f3d36674be8bd76f8f90a2f74b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:45.877487 1225 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_e0696987af214f9fb46f9c76b529fca5" } modify_external_catalogs: true
W20250623 08:04:45.878062 1226 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbc25f3d36674be8bd76f8f90a2f74b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:45.879395 1235 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_20d45edbd83549339391048931491a56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:45.880025 1235 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20d45edbd83549339391048931491a56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:45.880853 342 ts_tablet_manager.cc:1428] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250623 08:04:45.881304 1245 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_58ecf0b93bc44862912280870cdcd018"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:45.881901 1245 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58ecf0b93bc44862912280870cdcd018 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:45.883872 1248 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_b764664d1bdc4441907690b6616f3570" } new_table_name: "default.Table_4b49e65313cd47e5b682a79f0394d475" modify_external_catalogs: true
W20250623 08:04:45.859553 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 414) could send its response
I20250623 08:04:45.885349 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "077bb090b3ea4d029463304d7a6feee5" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:45.889412 2210 consensus_queue.cc:1035] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:45.889588 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "077bb090b3ea4d029463304d7a6feee5" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.889816 32516 raft_consensus.cc:2391] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:45.891268 343 ts_tablet_manager.cc:1428] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20250623 08:04:45.891620 32428 heartbeater.cc:499] Master 127.31.69.61:33313 was elected leader, sending a full tablet report...
I20250623 08:04:45.891986 343 tablet_bootstrap.cc:492] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.893884 343 tablet_bootstrap.cc:654] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.896281 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.897257 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23928d64bcef4c06af81698e688b7f94" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:45.898136 32645 raft_consensus.cc:2391] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:45.898566 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b61b02cf2904404f8c0ffef7d268fefc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.898772 32516 raft_consensus.cc:3058] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.901656 32516 raft_consensus.cc:2466] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.903961 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6857437f85ab40f6a24e74ce3b5efbdc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.904237 32516 raft_consensus.cc:3058] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.907310 981 ts_tablet_manager.cc:1918] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:45.907630 32516 raft_consensus.cc:2466] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.908576 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea588a48e6164d3db148fdaa306268f6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.908732 32514 raft_consensus.cc:3058] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.908885 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baccec88aefb4056be9578bca13f1591" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.909018 32513 raft_consensus.cc:2466] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.913012 32514 raft_consensus.cc:2466] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.917448 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f5f58f55070344ff98a3808f70e0c755" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.917645 32519 raft_consensus.cc:2466] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.917989 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7b9e843ff1344a1aaacf038f15f64cd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.918546 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8911308855cb4414be426dbcec592f4a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.918695 32516 raft_consensus.cc:2466] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.918735 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4cf1f3943a3d49318c435921ebeaaf33" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:45.918888 32513 raft_consensus.cc:3058] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.921756 32513 raft_consensus.cc:2466] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:45.922170 342 tablet_bootstrap.cc:492] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:45.922840 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 373) took 5967 ms. Trace:
I20250623 08:04:45.923856 342 tablet_bootstrap.cc:654] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.934455 952 rpcz_store.cc:276] 0623 08:04:39.955606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.955671 (+ 65us) service_pool.cc:225] Handling call
0623 08:04:45.621091 (+5665420us) ts_tablet_manager.cc:1145] Deleting tablet 9dbbdbebdf3b4b50bda31a94fc0de36f
0623 08:04:45.621104 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.758170 (+137066us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.871177 (+113007us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.922817 (+ 51640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5665225}
I20250623 08:04:45.941082 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 374) took 5982 ms. Trace:
I20250623 08:04:45.945461 2338 raft_consensus.cc:491] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.945650 2338 raft_consensus.cc:513] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.946390 2338 leader_election.cc:290] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.949726 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cd316e6f4447be958611ab07ed31fc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.950693 32510 raft_consensus.cc:2466] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:45.950160 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "185c2bf7b9254947bb76b7fe0924559e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.951400 952 rpcz_store.cc:276] 0623 08:04:39.958899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:39.958954 (+ 55us) service_pool.cc:225] Handling call
0623 08:04:45.940905 (+5981951us) ts_tablet_manager.cc:1145] Deleting tablet 30dfa986dfd44658a0ffb07796030520
0623 08:04:45.940922 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.941057 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5981756}
I20250623 08:04:45.916965 2229 raft_consensus.cc:491] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.908447 981 log.cc:1199] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/adc3cf4355c24b7395fcb22af8520346
I20250623 08:04:45.955695 2229 raft_consensus.cc:513] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.955971 981 ts_tablet_manager.cc:1939] T adc3cf4355c24b7395fcb22af8520346 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:45.956266 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 423) took 3966 ms. Trace:
I20250623 08:04:45.956292 2229 leader_election.cc:290] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.956346 981 rpcz_store.cc:276] 0623 08:04:41.989766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990232 (+ 466us) service_pool.cc:225] Handling call
0623 08:04:45.841351 (+3851119us) ts_tablet_manager.cc:1145] Deleting tablet adc3cf4355c24b7395fcb22af8520346
0623 08:04:45.841360 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.845670 (+ 4310us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.907193 (+ 61523us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.956250 (+ 49057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3751249}
I20250623 08:04:45.939371 1682 raft_consensus.cc:491] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:45.956656 981 tablet_replica.cc:331] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.956851 981 raft_consensus.cc:2241] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.956643 1682 raft_consensus.cc:513] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:45.957000 981 raft_consensus.cc:2270] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.957412 1682 leader_election.cc:290] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:45.947759 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e06ef06db2643779f93617e27f100dd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:45.958595 32589 leader_election.cc:343] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:45.910401 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 420) could send its response
W20250623 08:04:45.959059 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 421) could send its response
W20250623 08:04:45.959224 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 409) could send its response
W20250623 08:04:45.959378 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 422) could send its response
W20250623 08:04:45.959527 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 427) could send its response
I20250623 08:04:45.960997 32314 leader_election.cc:304] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:45.961439 1682 raft_consensus.cc:3058] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:45.961894 981 ts_tablet_manager.cc:1905] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.962862 343 tablet_bootstrap.cc:492] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.962991 343 ts_tablet_manager.cc:1397] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.005s
I20250623 08:04:45.963686 343 raft_consensus.cc:357] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.963856 343 raft_consensus.cc:383] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.963915 343 raft_consensus.cc:738] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.963990 1682 raft_consensus.cc:2747] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:45.964145 343 consensus_queue.cc:260] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.965834 343 ts_tablet_manager.cc:1428] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
W20250623 08:04:45.965713 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 423) could send its response
I20250623 08:04:45.967489 343 tablet_bootstrap.cc:492] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:45.969640 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea77b97018b74708bd6057bd2d65f90b" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.970321 1682 raft_consensus.cc:491] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:45.970480 1682 raft_consensus.cc:513] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.971082 1682 leader_election.cc:290] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:45.974022 343 tablet_bootstrap.cc:654] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:45.977629 981 ts_tablet_manager.cc:1918] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:45.981745 981 log.cc:1199] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/db81250d1e164a7d94affd764fc0ff0c
I20250623 08:04:45.982499 981 ts_tablet_manager.cc:1939] T db81250d1e164a7d94affd764fc0ff0c P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:45.983701 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 424) took 3993 ms. Trace:
I20250623 08:04:45.983870 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cac83131e6854890b803b3d89ded7cf8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:45.985138 981 rpcz_store.cc:276] 0623 08:04:41.989838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990270 (+ 432us) service_pool.cc:225] Handling call
0623 08:04:45.956609 (+3966339us) ts_tablet_manager.cc:1145] Deleting tablet db81250d1e164a7d94affd764fc0ff0c
0623 08:04:45.956619 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.970220 (+ 13601us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.977497 (+ 7277us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:45.983683 (+ 6186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3866428}
I20250623 08:04:45.986023 981 tablet_replica.cc:331] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:45.986672 981 raft_consensus.cc:2241] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:45.987331 981 raft_consensus.cc:2270] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.988241 981 ts_tablet_manager.cc:1905] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:45.955049 342 tablet_bootstrap.cc:492] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:45.997174 342 ts_tablet_manager.cc:1397] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.000s
I20250623 08:04:45.997267 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 396) took 5291 ms. Trace:
I20250623 08:04:45.997414 952 rpcz_store.cc:276] 0623 08:04:40.706179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.710228 (+ 4049us) service_pool.cc:225] Handling call
0623 08:04:45.997127 (+5286899us) ts_tablet_manager.cc:1145] Deleting tablet a383d10d53b04d5699dd3c4bad8878c7
0623 08:04:45.997139 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:45.997250 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5286607}
I20250623 08:04:45.997746 952 tablet_replica.cc:331] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:45.997922 952 raft_consensus.cc:2241] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:45.997968 342 raft_consensus.cc:357] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.998095 342 raft_consensus.cc:383] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:45.998136 342 raft_consensus.cc:738] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:45.998193 952 raft_consensus.cc:2270] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:45.998318 342 consensus_queue.cc:260] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:45.998929 342 ts_tablet_manager.cc:1428] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:45.999233 342 tablet_bootstrap.cc:492] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:45.999405 343 tablet_bootstrap.cc:492] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:45.999547 343 ts_tablet_manager.cc:1397] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.007s
I20250623 08:04:46.000223 343 raft_consensus.cc:357] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.000367 343 raft_consensus.cc:383] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.000408 343 raft_consensus.cc:738] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.000594 343 consensus_queue.cc:260] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.001026 343 ts_tablet_manager.cc:1428] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:45.965356 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cac83131e6854890b803b3d89ded7cf8" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.968906 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea77b97018b74708bd6057bd2d65f90b" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:45.985574 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e06ef06db2643779f93617e27f100dd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.003896 32513 raft_consensus.cc:2466] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:46.005051 343 tablet_bootstrap.cc:492] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.006618 343 tablet_bootstrap.cc:654] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.014788 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8911308855cb4414be426dbcec592f4a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:46.015123 32645 raft_consensus.cc:2466] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:46.016754 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 424) could send its response
W20250623 08:04:46.017617 32314 leader_election.cc:343] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 789d1c20aa72480ab71123c799bbde4c
I20250623 08:04:46.018940 343 tablet_bootstrap.cc:492] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:46.019119 343 ts_tablet_manager.cc:1397] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20250623 08:04:46.019883 343 raft_consensus.cc:357] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.020076 343 raft_consensus.cc:383] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.020141 343 raft_consensus.cc:738] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.020339 343 consensus_queue.cc:260] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.020813 343 ts_tablet_manager.cc:1428] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:46.021087 343 tablet_bootstrap.cc:492] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.022820 343 tablet_bootstrap.cc:654] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.032373 32589 leader_election.cc:304] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.032748 2338 raft_consensus.cc:2747] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.035017 343 tablet_bootstrap.cc:492] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:46.035172 343 ts_tablet_manager.cc:1397] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20250623 08:04:46.035910 343 raft_consensus.cc:357] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.036087 343 raft_consensus.cc:383] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.036160 343 raft_consensus.cc:738] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.036358 343 consensus_queue.cc:260] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.036944 343 ts_tablet_manager.cc:1428] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:46.037209 343 tablet_bootstrap.cc:492] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.038815 343 tablet_bootstrap.cc:654] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.046896 2229 raft_consensus.cc:491] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.047050 2229 raft_consensus.cc:513] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.047622 2229 leader_election.cc:290] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:46.048628 339 tablet_bootstrap.cc:492] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:46.049396 343 tablet_bootstrap.cc:492] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:46.049571 343 ts_tablet_manager.cc:1397] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20250623 08:04:46.050251 343 raft_consensus.cc:357] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.050448 343 raft_consensus.cc:383] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.050522 343 raft_consensus.cc:738] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.050726 343 consensus_queue.cc:260] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.051209 32589 leader_election.cc:304] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.051296 343 ts_tablet_manager.cc:1428] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:46.051505 2336 raft_consensus.cc:2802] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.051618 2336 raft_consensus.cc:695] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:46.051723 952 ts_tablet_manager.cc:1905] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:46.051872 2336 consensus_queue.cc:237] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.054805 32589 leader_election.cc:304] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.055068 2210 raft_consensus.cc:2802] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.055190 2210 raft_consensus.cc:695] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:46.055325 2337 consensus_queue.cc:1035] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.055398 2210 consensus_queue.cc:237] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.055845 32589 leader_election.cc:304] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.056037 2337 raft_consensus.cc:2802] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:46.056102 2337 raft_consensus.cc:491] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.056170 2337 raft_consensus.cc:3058] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.051614 343 tablet_bootstrap.cc:492] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.058324 343 tablet_bootstrap.cc:654] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.060599 2337 raft_consensus.cc:513] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.061342 2337 leader_election.cc:290] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.066373 339 ts_tablet_manager.cc:1397] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.227s user 0.012s sys 0.012s
I20250623 08:04:46.067171 2336 consensus_queue.cc:1035] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.046896 32314 leader_election.cc:304] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:46.067431 343 tablet_bootstrap.cc:492] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:46.067515 2229 raft_consensus.cc:2747] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.067620 343 ts_tablet_manager.cc:1397] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
W20250623 08:04:46.067790 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 48757.3us
I20250623 08:04:46.068017 32314 leader_election.cc:304] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:46.068219 2229 raft_consensus.cc:2747] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.068403 343 raft_consensus.cc:357] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.068563 343 raft_consensus.cc:383] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.068282 32589 leader_election.cc:304] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:46.068645 32314 leader_election.cc:343] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 789d1c20aa72480ab71123c799bbde4c
I20250623 08:04:46.068748 32314 leader_election.cc:304] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:46.068632 343 raft_consensus.cc:738] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
W20250623 08:04:46.069129 32314 raft_consensus.cc:2684] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250623 08:04:46.069126 2336 raft_consensus.cc:2802] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.069293 2336 raft_consensus.cc:695] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
W20250623 08:04:46.069545 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 373) could send its response
I20250623 08:04:46.069942 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0107b41d5c04a5c818f6792f2c18ce5" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.070173 32518 raft_consensus.cc:2391] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:46.063894 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baccec88aefb4056be9578bca13f1591" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:46.072665 2336 consensus_queue.cc:237] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:46.074355 32589 leader_election.cc:343] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 2c5ecd4fa5574a1b90b6e82f89ee88f7
I20250623 08:04:46.075205 32589 leader_election.cc:304] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.074862 343 consensus_queue.cc:260] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.075594 2337 raft_consensus.cc:2747] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:46.076835 343 ts_tablet_manager.cc:1428] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250623 08:04:46.077524 343 tablet_bootstrap.cc:492] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.077200 32589 leader_election.cc:304] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.077874 2337 raft_consensus.cc:491] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.078759 2337 raft_consensus.cc:513] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.079552 2337 leader_election.cc:290] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.079833 2337 raft_consensus.cc:2802] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:46.079955 2337 raft_consensus.cc:491] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.080082 2337 raft_consensus.cc:3058] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.081322 2229 raft_consensus.cc:491] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:46.081476 2229 raft_consensus.cc:513] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.082024 2229 leader_election.cc:290] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.083259 2337 raft_consensus.cc:513] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.083902 2337 leader_election.cc:290] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.084167 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d0107b41d5c04a5c818f6792f2c18ce5" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:46.085037 32589 leader_election.cc:343] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.086405 32453 leader_election.cc:343] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.085271 2210 consensus_queue.cc:1035] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.087498 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8911308855cb4414be426dbcec592f4a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.086725 32589 leader_election.cc:304] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.088336 2210 raft_consensus.cc:2747] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.067198 339 raft_consensus.cc:357] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.088654 339 raft_consensus.cc:383] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.088732 339 raft_consensus.cc:738] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.088953 339 consensus_queue.cc:260] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:46.089035 32453 leader_election.cc:343] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.089802 339 ts_tablet_manager.cc:1428] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250623 08:04:46.090096 339 tablet_bootstrap.cc:492] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:46.090302 32453 leader_election.cc:343] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.090711 32453 consensus_peers.cc:487] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.091204 2014 consensus_queue.cc:1035] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.092440 343 tablet_bootstrap.cc:654] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.092684 339 tablet_bootstrap.cc:654] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.101706 32453 leader_election.cc:304] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.091113 32589 leader_election.cc:304] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.103567 32589 leader_election.cc:304] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.104287 2337 raft_consensus.cc:2802] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:46.104415 2337 raft_consensus.cc:491] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.104553 2337 raft_consensus.cc:3058] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.105273 2338 raft_consensus.cc:2802] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.105387 2338 raft_consensus.cc:695] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:46.105616 2338 consensus_queue.cc:237] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.107308 2337 raft_consensus.cc:513] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.107977 2337 leader_election.cc:290] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.108795 2160 raft_consensus.cc:2802] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.108934 2160 raft_consensus.cc:695] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:46.109162 2160 consensus_queue.cc:237] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.118559 1957 consensus_queue.cc:1035] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.108438 32589 leader_election.cc:304] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.111244 2338 raft_consensus.cc:491] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.120025 2338 raft_consensus.cc:513] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:46.120283 32589 leader_election.cc:343] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.120393 32589 leader_election.cc:304] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.120568 2338 leader_election.cc:290] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.120699 2338 raft_consensus.cc:2802] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:46.120762 2338 raft_consensus.cc:491] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.120813 2338 raft_consensus.cc:3058] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.118580 32453 leader_election.cc:304] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.114842 2210 raft_consensus.cc:491] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.121780 2210 raft_consensus.cc:513] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.122159 2210 leader_election.cc:290] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.122387 1957 raft_consensus.cc:2762] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250623 08:04:46.122869 32453 leader_election.cc:304] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.123111 1957 raft_consensus.cc:2802] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.123211 1957 raft_consensus.cc:695] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:46.123463 1957 consensus_queue.cc:237] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.123893 2338 raft_consensus.cc:513] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.124332 2338 leader_election.cc:290] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.125157 343 tablet_bootstrap.cc:492] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:46.125711 2242 consensus_queue.cc:1035] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.125298 2329 raft_consensus.cc:2747] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.126163 343 ts_tablet_manager.cc:1397] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20250623 08:04:46.127038 343 raft_consensus.cc:357] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.127297 343 raft_consensus.cc:383] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.127379 343 raft_consensus.cc:738] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.127583 343 consensus_queue.cc:260] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.128064 343 ts_tablet_manager.cc:1428] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:46.128430 343 tablet_bootstrap.cc:492] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.130450 343 tablet_bootstrap.cc:654] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.131264 32453 leader_election.cc:304] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.131546 1627 raft_consensus.cc:2802] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.131675 1627 raft_consensus.cc:695] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:46.131937 1627 consensus_queue.cc:237] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.132776 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d79c96bd1c4ddfb4a1dd8a9d83c3fe" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.132905 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8911308855cb4414be426dbcec592f4a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:46.133158 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a24019b797467e857b744d701fb1ed" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.133162 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cd316e6f4447be958611ab07ed31fc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:46.133471 32377 raft_consensus.cc:3058] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.133697 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f5f58f55070344ff98a3808f70e0c755" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:46.133677 32589 leader_election.cc:343] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.134361 32589 leader_election.cc:343] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.134562 32589 leader_election.cc:343] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.135936 32378 raft_consensus.cc:3058] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.136386 32377 raft_consensus.cc:2466] T d5cd316e6f4447be958611ab07ed31fc P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:46.138056 32453 leader_election.cc:304] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:46.139015 1953 raft_consensus.cc:2802] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:46.139109 1953 raft_consensus.cc:491] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.139180 1953 raft_consensus.cc:3058] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:46.140666 32589 leader_election.cc:343] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.140864 32589 leader_election.cc:343] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.141703 1953 raft_consensus.cc:513] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.142138 32589 leader_election.cc:304] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:46.142464 1953 leader_election.cc:290] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:46.142622 2242 raft_consensus.cc:2802] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.142735 2242 raft_consensus.cc:695] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:46.142733 2397 raft_consensus.cc:491] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.142943 2397 raft_consensus.cc:513] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.143007 2242 consensus_queue.cc:237] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.143613 2397 leader_election.cc:290] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.144280 2242 consensus_queue.cc:1035] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.145622 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b213c8c685c43868df1e814c79fc61f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.145998 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "297bdf6c0e424190a35b55cbeefe7161" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:46.148711 32589 leader_election.cc:343] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.148900 32589 leader_election.cc:343] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.150774 2242 consensus_queue.cc:1035] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:46.152652 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 374) could send its response
W20250623 08:04:46.155628 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 396) could send its response
I20250623 08:04:46.156134 32589 leader_election.cc:304] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.156417 2242 raft_consensus.cc:2802] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:46.156499 2242 raft_consensus.cc:491] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.156553 2242 raft_consensus.cc:3058] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.159229 2242 raft_consensus.cc:513] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.159801 2242 leader_election.cc:290] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.159921 2242 raft_consensus.cc:491] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.160002 2242 raft_consensus.cc:513] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.160445 2242 leader_election.cc:290] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.161115 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e06ef06db2643779f93617e27f100dd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:46.161160 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a383d10d53b04d5699dd3c4bad8878c7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:46.161739 32589 leader_election.cc:343] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.162319 32589 leader_election.cc:343] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.174080 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8911308855cb4414be426dbcec592f4a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:46.174382 32649 raft_consensus.cc:2391] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:46.178453 1823 raft_consensus.cc:491] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:46.178646 1823 raft_consensus.cc:513] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.179361 1823 leader_election.cc:290] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:46.182341 343 tablet_bootstrap.cc:492] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:46.182691 343 ts_tablet_manager.cc:1397] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.003s
I20250623 08:04:46.184010 343 raft_consensus.cc:357] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.188302 343 raft_consensus.cc:383] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.189028 343 raft_consensus.cc:738] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.189738 343 consensus_queue.cc:260] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.193883 32378 raft_consensus.cc:3058] T ea77b97018b74708bd6057bd2d65f90b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.195516 32374 raft_consensus.cc:3058] T cac83131e6854890b803b3d89ded7cf8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.198225 343 ts_tablet_manager.cc:1428] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250623 08:04:46.198580 343 tablet_bootstrap.cc:492] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.200776 343 tablet_bootstrap.cc:654] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.208974 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baccec88aefb4056be9578bca13f1591" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:46.209151 32510 raft_consensus.cc:3058] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.212911 32510 raft_consensus.cc:2466] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:46.216295 1242 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet dcfe68d4e6ae470d816ac4c494dcaa51
I20250623 08:04:46.218307 1242 catalog_manager.cc:6652] ReplaceTablet: tablet dcfe68d4e6ae470d816ac4c494dcaa51 deleted and replaced by tablet 06fe16d2aa1c4a059e91348c40410bc2
I20250623 08:04:46.239710 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f5f58f55070344ff98a3808f70e0c755" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:46.239943 32515 raft_consensus.cc:3058] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.239930 32511 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cd316e6f4447be958611ab07ed31fc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:46.240307 32511 raft_consensus.cc:3058] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.240978 339 tablet_bootstrap.cc:492] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:46.241150 339 ts_tablet_manager.cc:1397] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.151s user 0.003s sys 0.000s
I20250623 08:04:46.241755 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8911308855cb4414be426dbcec592f4a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:46.241967 32516 raft_consensus.cc:2391] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:46.242887 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d79c96bd1c4ddfb4a1dd8a9d83c3fe" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.239903 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a24019b797467e857b744d701fb1ed" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.244556 32511 raft_consensus.cc:2466] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:46.246312 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b213c8c685c43868df1e814c79fc61f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.246312 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e06ef06db2643779f93617e27f100dd" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:46.246681 32516 raft_consensus.cc:3058] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:46.247984 32589 consensus_peers.cc:487] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.248831 339 raft_consensus.cc:357] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.249553 32516 raft_consensus.cc:2466] T 4e06ef06db2643779f93617e27f100dd P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:46.250236 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a383d10d53b04d5699dd3c4bad8878c7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.251596 2397 consensus_queue.cc:1035] T d0107b41d5c04a5c818f6792f2c18ce5 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.252526 339 raft_consensus.cc:383] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.253183 339 raft_consensus.cc:738] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.254405 32515 raft_consensus.cc:2466] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:46.254422 339 consensus_queue.cc:260] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.256783 339 ts_tablet_manager.cc:1428] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250623 08:04:46.259928 32589 leader_election.cc:304] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.260308 2397 raft_consensus.cc:2802] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.260445 2397 raft_consensus.cc:695] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:46.260810 2397 consensus_queue.cc:237] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.262687 2242 consensus_queue.cc:1035] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.263111 339 tablet_bootstrap.cc:492] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:46.265605 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "297bdf6c0e424190a35b55cbeefe7161" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.266767 32589 leader_election.cc:304] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.267083 2242 raft_consensus.cc:2747] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250623 08:04:46.269399 32589 leader_election.cc:343] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.269587 32589 leader_election.cc:304] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.269891 2242 raft_consensus.cc:2747] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:46.270488 32589 leader_election.cc:343] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.270614 32589 leader_election.cc:304] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.271287 2337 raft_consensus.cc:2747] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:46.274413 32589 consensus_peers.cc:487] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f97f14cbaffb46d681d8b11c0213cae4. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:46.275048 32589 leader_election.cc:343] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.275161 32589 leader_election.cc:304] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.275393 2242 raft_consensus.cc:2747] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.276190 2242 consensus_queue.cc:1035] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.280114 1953 raft_consensus.cc:491] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.280288 1953 raft_consensus.cc:513] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.280802 339 tablet_bootstrap.cc:654] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.280912 1953 leader_election.cc:290] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:46.292176 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8de2b1105564fdab386442b1de4cedb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.292451 32513 raft_consensus.cc:2466] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
W20250623 08:04:46.292912 32589 leader_election.cc:343] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.293061 32589 leader_election.cc:304] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.293452 32589 leader_election.cc:304] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.293774 32589 leader_election.cc:304] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:46.294229 32589 leader_election.cc:343] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.294343 32589 leader_election.cc:304] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.294848 2242 raft_consensus.cc:2747] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.295089 2242 raft_consensus.cc:2802] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.295226 2242 raft_consensus.cc:695] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:46.295584 2242 consensus_queue.cc:237] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.289001 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8911308855cb4414be426dbcec592f4a" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:46.289880 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8de2b1105564fdab386442b1de4cedb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.296334 32374 raft_consensus.cc:2466] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:46.296689 2337 raft_consensus.cc:2802] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.296866 2337 raft_consensus.cc:695] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:46.297180 2337 consensus_queue.cc:237] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.298434 2336 raft_consensus.cc:2747] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:46.301708 32589 consensus_peers.cc:487] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db81250d1e164a7d94affd764fc0ff0c. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.303086 2397 raft_consensus.cc:491] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.303293 2397 raft_consensus.cc:513] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.303928 2397 leader_election.cc:290] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.304668 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7f759b707ef46df99873b92c694ad6b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:46.305234 32589 leader_election.cc:343] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.306398 32517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7f759b707ef46df99873b92c694ad6b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.306595 32517 raft_consensus.cc:2466] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
W20250623 08:04:46.306882 32589 consensus_peers.cc:487] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9dbbdbebdf3b4b50bda31a94fc0de36f. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.307189 32589 leader_election.cc:304] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.307538 2397 raft_consensus.cc:2802] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:46.307632 2397 raft_consensus.cc:491] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.307718 2397 raft_consensus.cc:3058] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.310446 2397 raft_consensus.cc:513] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.313423 981 ts_tablet_manager.cc:1918] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:46.313612 981 log.cc:1199] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/789d1c20aa72480ab71123c799bbde4c
I20250623 08:04:46.313710 2397 leader_election.cc:290] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.320237 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7f759b707ef46df99873b92c694ad6b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:46.320458 32515 raft_consensus.cc:3058] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.324023 32515 raft_consensus.cc:2466] T c7f759b707ef46df99873b92c694ad6b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:46.326444 32589 leader_election.cc:304] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:46.326908 2337 raft_consensus.cc:2802] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.327088 2337 raft_consensus.cc:695] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:46.333956 2337 consensus_queue.cc:237] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.336506 2242 raft_consensus.cc:491] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.336740 2242 raft_consensus.cc:513] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.337498 2242 leader_election.cc:290] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.342479 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74c1cdc83ffd4a09a163479ac58fd405" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:46.347460 32589 leader_election.cc:343] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.353018 2397 raft_consensus.cc:491] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.353204 2397 raft_consensus.cc:513] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.353901 2397 leader_election.cc:290] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.356834 2397 consensus_queue.cc:1035] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.359477 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e13dc85076b34903b35310b14dc31085" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:46.361946 32589 leader_election.cc:343] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.362741 2397 raft_consensus.cc:491] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.363001 2397 raft_consensus.cc:513] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.363729 2397 leader_election.cc:290] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.364807 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:46.365370 32589 leader_election.cc:343] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.374352 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b720fdea539f423ba2d2aa9f89b09ae9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:46.374647 32651 raft_consensus.cc:2466] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:46.377604 2188 raft_consensus.cc:491] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.377780 2188 raft_consensus.cc:513] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.378417 2188 leader_election.cc:290] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.386698 2407 consensus_queue.cc:1035] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:46.387188 32453 leader_election.cc:343] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: a8de2b1105564fdab386442b1de4cedb
W20250623 08:04:46.389618 32453 consensus_peers.cc:487] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa398354a9e44633857a75c0bbba08b3. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:46.390000 32589 consensus_peers.cc:487] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.314513 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7f759b707ef46df99873b92c694ad6b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:46.318071 339 tablet_bootstrap.cc:492] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:46.392545 339 ts_tablet_manager.cc:1397] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.132s user 0.002s sys 0.002s
I20250623 08:04:46.392645 2242 raft_consensus.cc:491] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7351aa31800245659edc9837e27daffd)
I20250623 08:04:46.392714 981 ts_tablet_manager.cc:1939] T 789d1c20aa72480ab71123c799bbde4c P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
W20250623 08:04:46.392728 32589 leader_election.cc:343] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.392817 2242 raft_consensus.cc:513] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.393362 2242 leader_election.cc:290] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.393339 339 raft_consensus.cc:357] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.393442 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 428) took 4403 ms. Trace:
I20250623 08:04:46.393507 339 raft_consensus.cc:383] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.393575 339 raft_consensus.cc:738] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.393752 339 consensus_queue.cc:260] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.389568 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecc73f52f0bc43c5bbdca2b89deecb21" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.341341 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74c1cdc83ffd4a09a163479ac58fd405" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.355614 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e13dc85076b34903b35310b14dc31085" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.355609 32377 raft_consensus.cc:3058] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.395946 339 ts_tablet_manager.cc:1428] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:46.396230 339 tablet_bootstrap.cc:492] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:46.364558 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.379886 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecc73f52f0bc43c5bbdca2b89deecb21" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.398671 339 tablet_bootstrap.cc:654] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:46.398849 32314 leader_election.cc:343] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: a8de2b1105564fdab386442b1de4cedb
I20250623 08:04:46.399369 32314 leader_election.cc:304] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:46.399639 2229 raft_consensus.cc:2747] T d0107b41d5c04a5c818f6792f2c18ce5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:46.401674 32453 leader_election.cc:343] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.402071 32453 consensus_peers.cc:487] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adc3cf4355c24b7395fcb22af8520346. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:46.411757 32589 leader_election.cc:343] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.412210 32589 leader_election.cc:343] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.412335 32589 leader_election.cc:304] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.412591 2329 raft_consensus.cc:2747] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:46.412750 32589 leader_election.cc:343] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.412874 32589 leader_election.cc:304] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.413118 2329 raft_consensus.cc:2747] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.418633 342 tablet_bootstrap.cc:654] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:46.429072 32589 leader_election.cc:343] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: f83f68ebda7e4826bbb17278cd4ac5a8
W20250623 08:04:46.429522 32589 leader_election.cc:343] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.429903 32589 leader_election.cc:304] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.430478 2242 raft_consensus.cc:2747] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:46.430693 32589 leader_election.cc:343] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.431871 32589 leader_election.cc:304] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.432484 2242 raft_consensus.cc:2747] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.440218 343 tablet_bootstrap.cc:492] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:46.440452 343 ts_tablet_manager.cc:1397] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.242s user 0.015s sys 0.005s
I20250623 08:04:46.441411 343 raft_consensus.cc:357] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.441664 343 raft_consensus.cc:383] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.441941 343 raft_consensus.cc:738] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.442251 343 consensus_queue.cc:260] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.442883 343 ts_tablet_manager.cc:1428] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:46.443290 343 tablet_bootstrap.cc:492] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.393505 981 rpcz_store.cc:276] 0623 08:04:41.990255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990298 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:45.985973 (+3995675us) ts_tablet_manager.cc:1145] Deleting tablet 789d1c20aa72480ab71123c799bbde4c
0623 08:04:45.985984 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.151547 (+165563us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.313301 (+161754us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.393427 (+ 80126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3895688}
I20250623 08:04:46.449846 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 426) took 4459 ms. Trace:
I20250623 08:04:46.449986 981 rpcz_store.cc:276] 0623 08:04:41.990061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990334 (+ 273us) service_pool.cc:225] Handling call
0623 08:04:46.449732 (+4459398us) ts_tablet_manager.cc:1145] Deleting tablet 79731106915d4cebb9dcc3c10386ffa9
0623 08:04:46.449743 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.449823 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4359351}
I20250623 08:04:46.450397 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 429) took 4460 ms. Trace:
I20250623 08:04:46.450512 981 rpcz_store.cc:276] 0623 08:04:41.990362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990538 (+ 176us) service_pool.cc:225] Handling call
0623 08:04:46.450285 (+4459747us) ts_tablet_manager.cc:1145] Deleting tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6
0623 08:04:46.450296 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.450381 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4359848}
I20250623 08:04:46.450886 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 430) took 4460 ms. Trace:
I20250623 08:04:46.450994 981 rpcz_store.cc:276] 0623 08:04:41.990500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990597 (+ 97us) service_pool.cc:225] Handling call
0623 08:04:46.450806 (+4460209us) ts_tablet_manager.cc:1145] Deleting tablet 70f9d21b8c144f9cada9bdbe0fbb1538
0623 08:04:46.450814 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.450873 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4360299}
I20250623 08:04:46.451458 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 431) took 4460 ms. Trace:
I20250623 08:04:46.451581 981 rpcz_store.cc:276] 0623 08:04:41.990597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990635 (+ 38us) service_pool.cc:225] Handling call
0623 08:04:46.451239 (+4460604us) ts_tablet_manager.cc:1145] Deleting tablet d0b2af698b6e4ed0ae66a9bb32be7b31
0623 08:04:46.451371 (+ 132us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.451440 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4360636}
I20250623 08:04:46.451917 981 tablet_replica.cc:331] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:46.452157 981 raft_consensus.cc:2241] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:46.465086 952 ts_tablet_manager.cc:1918] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:46.465299 952 log.cc:1199] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/4b11777e0bd04e61a5bc8bb6ec4d97c3
I20250623 08:04:46.465719 952 ts_tablet_manager.cc:1939] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:46.466104 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 397) took 5662 ms. Trace:
I20250623 08:04:46.466231 952 rpcz_store.cc:276] 0623 08:04:40.803706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.805598 (+ 1892us) service_pool.cc:225] Handling call
0623 08:04:45.997700 (+5192102us) ts_tablet_manager.cc:1145] Deleting tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3
0623 08:04:45.997709 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.208216 (+210507us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.464958 (+256742us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.466083 (+ 1125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5191937}
I20250623 08:04:46.466689 952 tablet_replica.cc:331] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:46.467082 952 raft_consensus.cc:2241] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:46.467303 952 raft_consensus.cc:2270] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:46.467947 952 ts_tablet_manager.cc:1905] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:46.484372 2197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250623 08:04:46.484918 2197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250623 08:04:46.485090 2197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250623 08:04:46.485198 2197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250623 08:04:46.495455 32375 raft_consensus.cc:3058] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:46.500465 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 100625us
I20250623 08:04:46.500813 32314 leader_election.cc:304] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
W20250623 08:04:46.501789 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 428) could send its response
W20250623 08:04:46.502089 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 426) could send its response
W20250623 08:04:46.502278 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 429) could send its response
W20250623 08:04:46.502471 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 430) could send its response
W20250623 08:04:46.502619 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 431) could send its response
I20250623 08:04:46.509320 339 tablet_bootstrap.cc:492] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:46.509720 339 ts_tablet_manager.cc:1397] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.114s user 0.014s sys 0.010s
W20250623 08:04:46.515393 1229 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 035bb06bc7dd42f681886def9d56eb73 consensus state has no leader
I20250623 08:04:46.527024 1682 raft_consensus.cc:2802] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:46.527135 1682 raft_consensus.cc:491] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:46.527204 1682 raft_consensus.cc:3058] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:46.529912 1682 raft_consensus.cc:513] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.530504 1682 leader_election.cc:290] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.531616 339 raft_consensus.cc:357] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.531765 339 raft_consensus.cc:383] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.531816 339 raft_consensus.cc:738] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.531975 339 consensus_queue.cc:260] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.532773 339 ts_tablet_manager.cc:1428] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250623 08:04:46.533308 339 tablet_bootstrap.cc:492] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:46.535570 339 tablet_bootstrap.cc:654] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.540674 981 raft_consensus.cc:2270] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:46.541559 981 ts_tablet_manager.cc:1905] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:46.559433 339 tablet_bootstrap.cc:492] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:46.559702 339 ts_tablet_manager.cc:1397] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.003s
I20250623 08:04:46.560501 339 raft_consensus.cc:357] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.561045 339 raft_consensus.cc:383] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.561147 339 raft_consensus.cc:738] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.561213 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b720fdea539f423ba2d2aa9f89b09ae9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.561358 339 consensus_queue.cc:260] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.561918 339 ts_tablet_manager.cc:1428] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:46.562217 339 tablet_bootstrap.cc:492] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:46.566586 339 tablet_bootstrap.cc:654] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.570593 1227 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet ea588a48e6164d3db148fdaa306268f6
I20250623 08:04:46.571543 981 ts_tablet_manager.cc:1918] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:46.571647 981 log.cc:1199] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/d19d468ce13d4108a9badc36fc503306
I20250623 08:04:46.571661 1225 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet bb06a340f82047ac8ec3516edb08f4b6
I20250623 08:04:46.572017 981 ts_tablet_manager.cc:1939] T d19d468ce13d4108a9badc36fc503306 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:46.575306 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 432) took 4584 ms. Trace:
I20250623 08:04:46.576177 981 rpcz_store.cc:276] 0623 08:04:41.990676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990820 (+ 144us) service_pool.cc:225] Handling call
0623 08:04:46.451862 (+4461042us) ts_tablet_manager.cc:1145] Deleting tablet d19d468ce13d4108a9badc36fc503306
0623 08:04:46.451872 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.564191 (+112319us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.571432 (+ 7241us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.575287 (+ 3855us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":88242,"tablet-delete.queue_time_us":4361152}
W20250623 08:04:46.577003 1218 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 035bb06bc7dd42f681886def9d56eb73 (table default.Table_53879d6f821f4167bf02ffbe967f73f7 [id=b3f407a7a26e4bd5ae9a719a5c4577c0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 035bb06bc7dd42f681886def9d56eb73 consensus state has no leader
I20250623 08:04:46.577376 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 433) took 4586 ms. Trace:
I20250623 08:04:46.578204 981 rpcz_store.cc:276] 0623 08:04:41.990786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990883 (+ 97us) service_pool.cc:225] Handling call
0623 08:04:46.576598 (+4585715us) ts_tablet_manager.cc:1145] Deleting tablet 52d0a356c74f4daea9093c998b4bf73e
0623 08:04:46.576609 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.576700 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4485719}
I20250623 08:04:46.583388 981 tablet_replica.cc:331] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:46.583828 981 raft_consensus.cc:2241] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:46.585311 339 tablet_bootstrap.cc:492] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:46.585491 339 ts_tablet_manager.cc:1397] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250623 08:04:46.586278 339 raft_consensus.cc:357] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.586490 339 raft_consensus.cc:383] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.586561 339 raft_consensus.cc:738] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.586823 339 consensus_queue.cc:260] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:46.586848 1213 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 67358.2us
I20250623 08:04:46.587378 339 ts_tablet_manager.cc:1428] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:46.587810 339 tablet_bootstrap.cc:492] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:46.506520 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 104044us
W20250623 08:04:46.590498 32453 consensus_peers.cc:487] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa398354a9e44633857a75c0bbba08b3. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.565927 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8de2b1105564fdab386442b1de4cedb" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
W20250623 08:04:46.590674 32453 consensus_peers.cc:487] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16331ba6a34942a5a1b4ff4ac7bac55e. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.590698 32513 raft_consensus.cc:3058] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
W20250623 08:04:46.591094 32453 leader_election.cc:343] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.591188 32453 leader_election.cc:304] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.591416 2160 raft_consensus.cc:2747] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.591586 32453 leader_election.cc:304] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:46.592774 32453 leader_election.cc:304] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:46.593777 1953 raft_consensus.cc:2802] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:46.593839 1953 raft_consensus.cc:491] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.593900 1953 raft_consensus.cc:3058] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.597149 1953 raft_consensus.cc:513] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.597899 1953 leader_election.cc:290] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:46.599815 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 397) could send its response
I20250623 08:04:46.602939 32513 raft_consensus.cc:2466] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 2.
I20250623 08:04:46.603065 1359 raft_consensus.cc:2762] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250623 08:04:46.605434 342 tablet_bootstrap.cc:492] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:46.610114 2337 consensus_queue.cc:1035] T 7dc5e0fbeba4412aa42d116cd8882238 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.610580 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet dcfe68d4e6ae470d816ac4c494dcaa51 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fe16d2aa1c4a059e91348c40410bc2) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:46.611268 2337 raft_consensus.cc:491] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.611454 2337 raft_consensus.cc:513] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.612267 2337 leader_election.cc:290] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.613337 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:46.614027 32589 leader_election.cc:343] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.624045 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b720fdea539f423ba2d2aa9f89b09ae9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:46.624301 32651 raft_consensus.cc:3058] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.627523 32651 raft_consensus.cc:2466] T b720fdea539f423ba2d2aa9f89b09ae9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:46.632350 32589 leader_election.cc:343] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: f83f68ebda7e4826bbb17278cd4ac5a8
I20250623 08:04:46.632483 32589 leader_election.cc:304] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.632871 2337 raft_consensus.cc:2747] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:46.639854 32359 tablet_service.cc:1515] Processing DeleteTablet for tablet dcfe68d4e6ae470d816ac4c494dcaa51 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fe16d2aa1c4a059e91348c40410bc2) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:46.640142 32357 tablet_service.cc:1515] Processing DeleteTablet for tablet ea588a48e6164d3db148fdaa306268f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:45 UTC) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:46.640326 32353 tablet_service.cc:1515] Processing DeleteTablet for tablet bb06a340f82047ac8ec3516edb08f4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:45 UTC) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:46.641093 343 tablet_bootstrap.cc:700] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3: Time spent opening tablet: real 0.197s user 0.000s sys 0.001s
I20250623 08:04:46.641456 343 tablet_bootstrap.cc:654] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.641438 981 raft_consensus.cc:2270] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:46.642059 2337 raft_consensus.cc:491] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.642225 2337 raft_consensus.cc:513] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.618207 32483 tablet_service.cc:1515] Processing DeleteTablet for tablet dcfe68d4e6ae470d816ac4c494dcaa51 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fe16d2aa1c4a059e91348c40410bc2) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:46.619014 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet ea588a48e6164d3db148fdaa306268f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:45 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:46.619536 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet bb06a340f82047ac8ec3516edb08f4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:45 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:46.622092 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.636966 2160 consensus_queue.cc:1035] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.642978 2337 leader_election.cc:290] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.637516 32453 leader_election.cc:304] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:46.643488 342 ts_tablet_manager.cc:1397] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.644s user 0.019s sys 0.000s
I20250623 08:04:46.643679 2160 raft_consensus.cc:2802] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.643651 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf753ede17f429b82612ad12e13185c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.643803 2160 raft_consensus.cc:695] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:46.644065 2160 consensus_queue.cc:237] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:46.644208 32589 leader_election.cc:343] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.650678 2446 raft_consensus.cc:491] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.651346 2446 raft_consensus.cc:513] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.652532 2446 leader_election.cc:290] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:46.659338 1248 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5af1dea3c6ca4f498ce6c039d72a14b7 consensus state has no leader
W20250623 08:04:46.659420 32589 leader_election.cc:343] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.659543 32589 leader_election.cc:304] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.659914 2337 raft_consensus.cc:2747] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.661581 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf753ede17f429b82612ad12e13185c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.675123 342 raft_consensus.cc:357] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.675433 342 raft_consensus.cc:383] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.675555 342 raft_consensus.cc:738] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
W20250623 08:04:46.675547 32589 leader_election.cc:343] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.675666 32589 leader_election.cc:304] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.675983 342 consensus_queue.cc:260] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.676443 2337 raft_consensus.cc:2747] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.676673 342 ts_tablet_manager.cc:1428] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.033s user 0.001s sys 0.000s
I20250623 08:04:46.677829 1245 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_811a9724f3234933a3f6352f67b17e73" } modify_external_catalogs: true
I20250623 08:04:46.678354 1245 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_811a9724f3234933a3f6352f67b17e73" } modify_external_catalogs: true
I20250623 08:04:46.679015 1248 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_71fd58f8442b4035bf100aa5edc058da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:46.664063 981 ts_tablet_manager.cc:1905] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:46.679646 1248 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71fd58f8442b4035bf100aa5edc058da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:46.680577 1226 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_69a26f395448447eb6d121e1f5520160"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:46.681057 1226 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69a26f395448447eb6d121e1f5520160 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:46.681304 1235 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_6867d69edd3b45a3b76992f22cf0207a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:46.671212 339 tablet_bootstrap.cc:654] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.677433 32368 raft_consensus.cc:3058] T 59015204545e436cb558fe4465a67c97 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.688331 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet ea588a48e6164d3db148fdaa306268f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:45 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:46.688877 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet bb06a340f82047ac8ec3516edb08f4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:45 UTC) from {username='slave'} at 127.0.0.1:42984
W20250623 08:04:46.689018 1235 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6867d69edd3b45a3b76992f22cf0207a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:46.689330 1225 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_0a60238bfb2e432d94faeeb3792b594a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:46.690325 1225 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a60238bfb2e432d94faeeb3792b594a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:46.690590 1227 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_1a1b8de8d0a645e9a48938956c562be9" } modify_external_catalogs: true
I20250623 08:04:46.696059 2455 tablet_replica.cc:331] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:46.696458 2455 raft_consensus.cc:2241] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:46.696899 2455 pending_rounds.cc:70] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: Trying to abort 1 pending ops.
I20250623 08:04:46.697057 2455 pending_rounds.cc:77] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7170727469356081152 op_type: NO_OP noop_request { }
I20250623 08:04:46.697240 2455 raft_consensus.cc:2887] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:46.697389 2455 raft_consensus.cc:2270] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:46.698298 2455 ts_tablet_manager.cc:1905] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:46.699602 1213 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet dcfe68d4e6ae470d816ac4c494dcaa51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcfe68d4e6ae470d816ac4c494dcaa51 already in progress: creating tablet
I20250623 08:04:46.702248 2407 raft_consensus.cc:491] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.702467 2407 raft_consensus.cc:513] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.703346 2407 leader_election.cc:290] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:46.704006 343 tablet_bootstrap.cc:492] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:46.704201 343 ts_tablet_manager.cc:1397] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.261s user 0.000s sys 0.004s
W20250623 08:04:46.704416 32453 leader_election.cc:343] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.704730 32453 consensus_peers.cc:487] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:46.704855 32453 consensus_peers.cc:487] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.705304 343 raft_consensus.cc:357] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.705549 343 raft_consensus.cc:383] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.705651 343 raft_consensus.cc:738] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.705945 343 consensus_queue.cc:260] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.706647 343 ts_tablet_manager.cc:1428] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:46.707121 343 tablet_bootstrap.cc:492] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.712188 1957 raft_consensus.cc:491] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.712329 1957 raft_consensus.cc:513] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.712899 1957 leader_election.cc:290] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:46.716818 32453 leader_election.cc:343] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 3740dd120745493688eeb4c1ac311dcb
I20250623 08:04:46.714447 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a383d10d53b04d5699dd3c4bad8878c7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:46.718581 32645 raft_consensus.cc:2466] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:46.716061 2014 raft_consensus.cc:491] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.718894 32453 leader_election.cc:304] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:46.719139 1957 raft_consensus.cc:2802] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:46.719214 1957 raft_consensus.cc:491] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.719280 1957 raft_consensus.cc:3058] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.719017 2014 raft_consensus.cc:513] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.720021 2014 leader_election.cc:290] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.721778 343 tablet_bootstrap.cc:654] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.721772 1957 raft_consensus.cc:513] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.722937 1957 leader_election.cc:290] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:46.723055 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c337a37fcb141bca87212317d750de8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:46.723678 32589 leader_election.cc:343] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.723845 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a383d10d53b04d5699dd3c4bad8878c7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:46.724005 32649 raft_consensus.cc:3058] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.726686 32649 raft_consensus.cc:2466] T a383d10d53b04d5699dd3c4bad8878c7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:46.727463 952 ts_tablet_manager.cc:1918] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:46.710536 2260 raft_consensus.cc:491] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 125745849172449ba4be8244d61fdcb3)
I20250623 08:04:46.727793 2260 raft_consensus.cc:513] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.728515 2260 leader_election.cc:290] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:46.728785 2014 raft_consensus.cc:491] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.728909 2014 raft_consensus.cc:513] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.729450 2014 leader_election.cc:290] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.694377 32314 leader_election.cc:304] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:46.729862 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886fc0fa39c54bddb25ac01f68d95805" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.729923 2260 raft_consensus.cc:2802] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:46.730034 2260 raft_consensus.cc:695] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
W20250623 08:04:46.730315 32589 leader_election.cc:343] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.730261 2260 consensus_queue.cc:237] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:46.731753 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 432) could send its response
W20250623 08:04:46.731940 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 433) could send its response
I20250623 08:04:46.733291 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23928d64bcef4c06af81698e688b7f94" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:46.733502 32649 raft_consensus.cc:2391] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:46.718449 339 tablet_bootstrap.cc:492] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:46.736765 339 ts_tablet_manager.cc:1397] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.149s user 0.004s sys 0.000s
I20250623 08:04:46.737205 1231 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_5cf87169ed0d4987b396195d518aa055" } modify_external_catalogs: true
I20250623 08:04:46.737406 1231 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_5cf87169ed0d4987b396195d518aa055" } modify_external_catalogs: true
I20250623 08:04:46.737708 339 raft_consensus.cc:357] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.737864 339 raft_consensus.cc:383] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.737915 339 raft_consensus.cc:738] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.738097 339 consensus_queue.cc:260] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.739339 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11bb55088ab24b94af3e509810890247" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:46.722573 2446 raft_consensus.cc:491] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.742574 2446 raft_consensus.cc:513] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.742918 1227 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_1a1b8de8d0a645e9a48938956c562be9" } modify_external_catalogs: true
I20250623 08:04:46.743191 2446 leader_election.cc:290] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:46.720484 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c337a37fcb141bca87212317d750de8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.738926 32377 raft_consensus.cc:3058] T a0b1287cf3b6459eafd032afc77c5824 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.744601 2197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.69.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.31.69.62:39465,127.31.69.61:33313,127.31.69.60:39691
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.69.62:39465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.69.62
--webserver_port=33855
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 23 Jun 2025 07:50:47 UTC on 24a791456cd2
build id 6728
ASAN enabled
I20250623 08:04:46.745188 339 ts_tablet_manager.cc:1428] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250623 08:04:46.745474 339 tablet_bootstrap.cc:492] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:46.746707 2197 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20250623 08:04:46.746976 32589 leader_election.cc:343] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.747088 32589 leader_election.cc:304] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.747494 2014 raft_consensus.cc:2747] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.748149 2014 consensus_queue.cc:1035] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.750201 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b720fdea539f423ba2d2aa9f89b09ae9" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:46.750926 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3740dd120745493688eeb4c1ac311dcb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.751075 32381 raft_consensus.cc:2466] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:46.751616 32314 leader_election.cc:343] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: a8de2b1105564fdab386442b1de4cedb
I20250623 08:04:46.751839 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a383d10d53b04d5699dd3c4bad8878c7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.753846 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a383d10d53b04d5699dd3c4bad8878c7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:46.755771 339 tablet_bootstrap.cc:654] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.760561 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c3dcdcaa2e14cb2bd02ebb8f1d0e159" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:46.760782 32649 raft_consensus.cc:2391] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:46.686340 981 ts_tablet_manager.cc:1918] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:46.761332 981 log.cc:1199] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/6926c49b787941d98c647543bd3b9ed5
I20250623 08:04:46.763231 981 ts_tablet_manager.cc:1939] T 6926c49b787941d98c647543bd3b9ed5 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:46.728020 32453 leader_election.cc:304] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:46.734231 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "23928d64bcef4c06af81698e688b7f94" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.763644 1953 raft_consensus.cc:2802] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:46.741580 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "11bb55088ab24b94af3e509810890247" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.764398 342 tablet_bootstrap.cc:492] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:46.766247 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886fc0fa39c54bddb25ac01f68d95805" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.767402 1953 raft_consensus.cc:695] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
W20250623 08:04:46.767825 32589 leader_election.cc:343] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.767828 1953 consensus_queue.cc:237] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.767990 32589 leader_election.cc:304] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.770213 2329 raft_consensus.cc:2747] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.771409 952 log.cc:1199] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/d5cd316e6f4447be958611ab07ed31fc
I20250623 08:04:46.771836 952 ts_tablet_manager.cc:1939] T d5cd316e6f4447be958611ab07ed31fc P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:46.772137 32453 leader_election.cc:343] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.772547 32453 leader_election.cc:304] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:46.772657 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 398) took 5904 ms. Trace:
I20250623 08:04:46.772845 1823 raft_consensus.cc:2802] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:46.772919 1823 raft_consensus.cc:491] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:46.772991 1823 raft_consensus.cc:3058] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
W20250623 08:04:46.773180 32453 leader_election.cc:343] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.773533 32453 leader_election.cc:343] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.775597 1823 raft_consensus.cc:513] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.776126 1823 leader_election.cc:290] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:46.778797 32453 leader_election.cc:343] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 3740dd120745493688eeb4c1ac311dcb
I20250623 08:04:46.779855 2197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250623 08:04:46.781889 342 tablet_bootstrap.cc:654] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.781877 2014 consensus_queue.cc:1035] T a0b1287cf3b6459eafd032afc77c5824 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.803579 2329 raft_consensus.cc:491] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.803763 2329 raft_consensus.cc:513] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.804253 2014 raft_consensus.cc:491] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.804384 2329 leader_election.cc:290] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.804816 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 435) took 4813 ms. Trace:
I20250623 08:04:46.804935 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f2a63b5a734c8f9b7a53361590245d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.804941 981 rpcz_store.cc:276] 0623 08:04:41.991019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.991058 (+ 39us) service_pool.cc:225] Handling call
0623 08:04:46.583323 (+4592265us) ts_tablet_manager.cc:1145] Deleting tablet 6926c49b787941d98c647543bd3b9ed5
0623 08:04:46.583335 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.682898 (+ 99563us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.686236 (+ 3338us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.804792 (+118556us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4492320}
I20250623 08:04:46.805076 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f2a63b5a734c8f9b7a53361590245d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:46.805327 32589 leader_election.cc:343] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:46.807202 32589 leader_election.cc:343] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.804373 2014 raft_consensus.cc:513] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.807299 32589 leader_election.cc:304] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.807469 2329 raft_consensus.cc:2747] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.808495 2014 leader_election.cc:290] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.772833 952 rpcz_store.cc:276] 0623 08:04:40.867759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:40.868567 (+ 808us) service_pool.cc:225] Handling call
0623 08:04:46.466646 (+5598079us) ts_tablet_manager.cc:1145] Deleting tablet d5cd316e6f4447be958611ab07ed31fc
0623 08:04:46.466655 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.665809 (+199154us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.727350 (+ 61541us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.772640 (+ 45290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5586114}
I20250623 08:04:46.808880 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3576624314d4e85b411aab9109683f7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:46.809078 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 398) could send its response
I20250623 08:04:46.809170 952 tablet_replica.cc:331] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: stopping tablet replica
W20250623 08:04:46.809374 32589 leader_election.cc:343] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.809424 952 raft_consensus.cc:2241] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:46.809756 952 pending_rounds.cc:70] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:46.809816 952 pending_rounds.cc:77] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7170727466227027968 op_type: NO_OP noop_request { }
I20250623 08:04:46.809899 952 raft_consensus.cc:2887] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:46.809978 952 raft_consensus.cc:2270] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:46.810511 952 ts_tablet_manager.cc:1905] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:46.771279 32314 leader_election.cc:343] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 6c3dcdcaa2e14cb2bd02ebb8f1d0e159
I20250623 08:04:46.813478 32314 leader_election.cc:304] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:46.813771 2260 raft_consensus.cc:3058] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.816599 1242 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:46.816704 2260 raft_consensus.cc:2747] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.817198 1242 catalog_manager.cc:5582] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:46.818120 1242 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]). Expected version 1 got 0
W20250623 08:04:46.818132 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 46855.8us
I20250623 08:04:46.818542 32314 leader_election.cc:304] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:46.819060 2446 raft_consensus.cc:3058] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.819310 981 tablet_replica.cc:331] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:46.819268 1229 catalog_manager.cc:5582] T 8111e6117c1541ab8905d5a5a7e37292 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:46.819533 981 raft_consensus.cc:2241] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:46.819682 981 raft_consensus.cc:2270] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:46.820220 981 ts_tablet_manager.cc:1905] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:46.821188 2446 raft_consensus.cc:2747] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.823032 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 0eec43a08c9c4211aa2cf53ff54b88f5 (DEFAULT_TABLE table=default.Table_14e9f0f5ff5d42d9bdecf7f91482d8b1 [id=eb2ada6a6890451cb5eb869c176eecec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.823199 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 0eec43a08c9c4211aa2cf53ff54b88f5 (DEFAULT_TABLE table=default.Table_14e9f0f5ff5d42d9bdecf7f91482d8b1 [id=eb2ada6a6890451cb5eb869c176eecec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.823406 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eec43a08c9c4211aa2cf53ff54b88f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.823596 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eec43a08c9c4211aa2cf53ff54b88f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.823637 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3576624314d4e85b411aab9109683f7" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:46.824507 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 0eec43a08c9c4211aa2cf53ff54b88f5 (DEFAULT_TABLE table=default.Table_14e9f0f5ff5d42d9bdecf7f91482d8b1 [id=eb2ada6a6890451cb5eb869c176eecec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.824888 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eec43a08c9c4211aa2cf53ff54b88f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:46.829586 32589 leader_election.cc:343] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.829717 32589 leader_election.cc:304] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.830363 2337 raft_consensus.cc:2747] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.831000 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 764244f33a1a48fdad410597429d3461 (DEFAULT_TABLE table=default.Table_1e293608b61743279ff20716c2953e24 [id=d7d9ed45091446fea8933157fd6e0db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.831403 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 764244f33a1a48fdad410597429d3461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.830850 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 764244f33a1a48fdad410597429d3461 (DEFAULT_TABLE table=default.Table_1e293608b61743279ff20716c2953e24 [id=d7d9ed45091446fea8933157fd6e0db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.831972 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 764244f33a1a48fdad410597429d3461 (DEFAULT_TABLE table=default.Table_1e293608b61743279ff20716c2953e24 [id=d7d9ed45091446fea8933157fd6e0db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.832324 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 764244f33a1a48fdad410597429d3461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.832368 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 764244f33a1a48fdad410597429d3461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.836122 343 tablet_bootstrap.cc:492] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:46.836298 343 ts_tablet_manager.cc:1397] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.129s user 0.002s sys 0.002s
I20250623 08:04:46.837435 343 raft_consensus.cc:357] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.837661 343 raft_consensus.cc:383] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.837765 343 raft_consensus.cc:738] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.838027 343 consensus_queue.cc:260] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.839612 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 0571fa96923144d5ac39429d252c8723 (DEFAULT_TABLE table=default.Table_1ef88ab1f1cd49139106aed3732b50a8 [id=90865844914743379da60c53e1c3a0d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.839743 32483 tablet_service.cc:1468] Processing CreateTablet for tablet 0571fa96923144d5ac39429d252c8723 (DEFAULT_TABLE table=default.Table_1ef88ab1f1cd49139106aed3732b50a8 [id=90865844914743379da60c53e1c3a0d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.840102 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0571fa96923144d5ac39429d252c8723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.819073 32314 leader_election.cc:304] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:46.843969 2446 raft_consensus.cc:2747] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 08:04:46.844878 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 435) could send its response
I20250623 08:04:46.847609 339 tablet_bootstrap.cc:492] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:46.847811 339 ts_tablet_manager.cc:1397] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.000s
I20250623 08:04:46.848767 339 raft_consensus.cc:357] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.849002 339 raft_consensus.cc:383] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.850143 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3740dd120745493688eeb4c1ac311dcb" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:46.850203 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0571fa96923144d5ac39429d252c8723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.850301 32381 raft_consensus.cc:3058] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:46.852972 32381 raft_consensus.cc:2466] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 2.
I20250623 08:04:46.853175 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 4597642b2a334baab0a336dace88de0f (DEFAULT_TABLE table=default.Table_cddb0cb673ab4aacb6e69f181369ca8a [id=8afa0fb6b40242fb826b52206a1173c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.853536 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4597642b2a334baab0a336dace88de0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.856258 2455 ts_tablet_manager.cc:1918] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:46.856411 2455 log.cc:1199] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/ea588a48e6164d3db148fdaa306268f6
I20250623 08:04:46.856793 2455 ts_tablet_manager.cc:1939] T ea588a48e6164d3db148fdaa306268f6 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:46.857657 2455 tablet_replica.cc:331] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:46.857834 2455 raft_consensus.cc:2241] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:46.857965 2455 raft_consensus.cc:2270] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:46.858417 2455 ts_tablet_manager.cc:1905] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:46.860746 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 4597642b2a334baab0a336dace88de0f (DEFAULT_TABLE table=default.Table_cddb0cb673ab4aacb6e69f181369ca8a [id=8afa0fb6b40242fb826b52206a1173c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.861176 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4597642b2a334baab0a336dace88de0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.849129 339 raft_consensus.cc:738] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.863062 339 consensus_queue.cc:260] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.863803 339 ts_tablet_manager.cc:1428] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250623 08:04:46.863891 343 ts_tablet_manager.cc:1428] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:46.867029 343 tablet_bootstrap.cc:492] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:46.868785 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 486f00b38c444ea9ba6f6482e05f873d (DEFAULT_TABLE table=default.Table_68f032ade7314d0da985fa8c94b11a8c [id=9895a12eef5a415cacd19cd277721b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.869163 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486f00b38c444ea9ba6f6482e05f873d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.872687 32636 tablet_service.cc:1468] Processing CreateTablet for tablet e259ef662ee44d1ca5081c05e9fed5f6 (DEFAULT_TABLE table=default.Table_bc37bf6911b5401c904eef106dde3592 [id=19ae0916b8c6480f93b92da98de7ec4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.873024 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e259ef662ee44d1ca5081c05e9fed5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.873971 2014 raft_consensus.cc:491] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.874105 2014 raft_consensus.cc:513] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.874627 2014 leader_election.cc:290] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.875548 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f080a682e532406a83712b6f3e6d2f7b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:46.876487 32589 leader_election.cc:343] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.876838 2329 raft_consensus.cc:491] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:46.876935 2329 raft_consensus.cc:513] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.877357 2329 leader_election.cc:290] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:46.878346 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:46.878870 32589 leader_election.cc:343] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.879472 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 486f00b38c444ea9ba6f6482e05f873d (DEFAULT_TABLE table=default.Table_68f032ade7314d0da985fa8c94b11a8c [id=9895a12eef5a415cacd19cd277721b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.879760 32453 leader_election.cc:304] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3
I20250623 08:04:46.880143 32489 tablet_service.cc:1468] Processing CreateTablet for tablet e259ef662ee44d1ca5081c05e9fed5f6 (DEFAULT_TABLE table=default.Table_bc37bf6911b5401c904eef106dde3592 [id=19ae0916b8c6480f93b92da98de7ec4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.880132 1953 raft_consensus.cc:2802] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:46.880636 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e259ef662ee44d1ca5081c05e9fed5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.875344 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f080a682e532406a83712b6f3e6d2f7b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.878026 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:46.880880 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486f00b38c444ea9ba6f6482e05f873d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:46.881793 32589 leader_election.cc:343] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.881844 339 tablet_bootstrap.cc:492] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:46.881877 32589 leader_election.cc:304] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:46.882216 32589 leader_election.cc:343] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:46.882292 32589 leader_election.cc:304] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:46.882660 2329 raft_consensus.cc:2747] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.882738 2242 raft_consensus.cc:2747] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:46.883850 339 tablet_bootstrap.cc:654] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.888331 32484 tablet_service.cc:1468] Processing CreateTablet for tablet ebe6148c11144fdb8519a3973e822159 (DEFAULT_TABLE table=default.Table_306d24f2f3484139b3586e0a1ebb8562 [id=2f72a0d782b54f109937d1ab340afd89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.888695 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebe6148c11144fdb8519a3973e822159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.892050 952 ts_tablet_manager.cc:1918] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250623 08:04:46.896075 343 tablet_bootstrap.cc:654] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.903846 981 ts_tablet_manager.cc:1918] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:46.904037 981 log.cc:1199] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/9bfa787f828b411ab68d81bb896f7fb6
W20250623 08:04:46.895985 32453 consensus_peers.cc:487] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.904523 981 ts_tablet_manager.cc:1939] T 9bfa787f828b411ab68d81bb896f7fb6 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:46.905478 32629 tablet_service.cc:1468] Processing CreateTablet for tablet ebe6148c11144fdb8519a3973e822159 (DEFAULT_TABLE table=default.Table_306d24f2f3484139b3586e0a1ebb8562 [id=2f72a0d782b54f109937d1ab340afd89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.905889 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebe6148c11144fdb8519a3973e822159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.905916 2337 consensus_queue.cc:1035] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.913479 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 51fce46407ac4e57a74ad696e85daac7 (DEFAULT_TABLE table=default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.914321 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51fce46407ac4e57a74ad696e85daac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.880882 1953 raft_consensus.cc:695] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:46.922847 2337 consensus_queue.cc:1035] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:46.924188 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:46.910319 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet ea588a48e6164d3db148fdaa306268f6 (table default.Table_b523bec7182443139e891357d295e3a1 [id=1c4298a5f9014d8eb4ccad8c8fbdeeac]) successfully deleted
W20250623 08:04:46.926277 2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:46.932760 32636 tablet_service.cc:1468] Processing CreateTablet for tablet fa41bab84b384c10ace3ce176698ebe4 (DEFAULT_TABLE table=default.Table_cace1e64be664c9c81b588e80d704aaa [id=7ed25123fbc749909d1d90134be64278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.933204 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa41bab84b384c10ace3ce176698ebe4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:46.936069 32453 consensus_peers.cc:487] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.938095 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 5930085e18d84823b27791a65750663f (DEFAULT_TABLE table=default.Table_1631659a168049fba9411207edd8b56f [id=10d3dacba1db466893ef47b1c42a8f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.938424 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5930085e18d84823b27791a65750663f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.940191 952 log.cc:1199] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/aa398354a9e44633857a75c0bbba08b3
I20250623 08:04:46.940613 952 ts_tablet_manager.cc:1939] T aa398354a9e44633857a75c0bbba08b3 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:46.940708 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 5930085e18d84823b27791a65750663f (DEFAULT_TABLE table=default.Table_1631659a168049fba9411207edd8b56f [id=10d3dacba1db466893ef47b1c42a8f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.940912 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 399) took 5729 ms. Trace:
I20250623 08:04:46.940973 952 rpcz_store.cc:276] 0623 08:04:41.211187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.211247 (+ 60us) service_pool.cc:225] Handling call
0623 08:04:46.809114 (+5597867us) ts_tablet_manager.cc:1145] Deleting tablet aa398354a9e44633857a75c0bbba08b3
0623 08:04:46.809126 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.889069 (+ 79943us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.891973 (+ 2904us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.940899 (+ 48926us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5597663}
W20250623 08:04:46.941215 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 399) could send its response
I20250623 08:04:46.941385 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5930085e18d84823b27791a65750663f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.941429 1953 consensus_queue.cc:237] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:46.942538 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 51fce46407ac4e57a74ad696e85daac7 (DEFAULT_TABLE table=default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.942906 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51fce46407ac4e57a74ad696e85daac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.947654 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 83e8e9739af0464787afc3580a034cd6 (DEFAULT_TABLE table=default.Table_43ff9ceffb4042b5ab1b041b76bdfc96 [id=2013d4281dc84c089dc17304aefd0837]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:46.948829 32589 consensus_peers.cc:487] T d5cd316e6f4447be958611ab07ed31fc P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5cd316e6f4447be958611ab07ed31fc. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:46.949656 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83e8e9739af0464787afc3580a034cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.953001 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 83e8e9739af0464787afc3580a034cd6 (DEFAULT_TABLE table=default.Table_43ff9ceffb4042b5ab1b041b76bdfc96 [id=2013d4281dc84c089dc17304aefd0837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.953338 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83e8e9739af0464787afc3580a034cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.958230 32483 tablet_service.cc:1468] Processing CreateTablet for tablet fa41bab84b384c10ace3ce176698ebe4 (DEFAULT_TABLE table=default.Table_cace1e64be664c9c81b588e80d704aaa [id=7ed25123fbc749909d1d90134be64278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.958606 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa41bab84b384c10ace3ce176698ebe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.962110 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 049b9194c9f74b8997c06469d74b9795 (DEFAULT_TABLE table=default.Table_4d2fc00363204f8485905af018f7911a [id=22f134d5d55f40ef98dec2f14a6d55ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.962489 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049b9194c9f74b8997c06469d74b9795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.966593 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 049b9194c9f74b8997c06469d74b9795 (DEFAULT_TABLE table=default.Table_4d2fc00363204f8485905af018f7911a [id=22f134d5d55f40ef98dec2f14a6d55ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.967020 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049b9194c9f74b8997c06469d74b9795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.908042 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 0571fa96923144d5ac39429d252c8723 (DEFAULT_TABLE table=default.Table_1ef88ab1f1cd49139106aed3732b50a8 [id=90865844914743379da60c53e1c3a0d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.908632 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 4597642b2a334baab0a336dace88de0f (DEFAULT_TABLE table=default.Table_cddb0cb673ab4aacb6e69f181369ca8a [id=8afa0fb6b40242fb826b52206a1173c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.908939 32359 tablet_service.cc:1468] Processing CreateTablet for tablet 486f00b38c444ea9ba6f6482e05f873d (DEFAULT_TABLE table=default.Table_68f032ade7314d0da985fa8c94b11a8c [id=9895a12eef5a415cacd19cd277721b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.968711 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0571fa96923144d5ac39429d252c8723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.968892 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4597642b2a334baab0a336dace88de0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.969163 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 425) took 4979 ms. Trace:
I20250623 08:04:46.969239 981 rpcz_store.cc:276] 0623 08:04:41.989934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.989995 (+ 61us) service_pool.cc:225] Handling call
0623 08:04:46.805601 (+4815606us) ts_tablet_manager.cc:1145] Deleting tablet 9bfa787f828b411ab68d81bb896f7fb6
0623 08:04:46.805617 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.885365 (+ 79748us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.903725 (+ 18360us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:46.969144 (+ 65419us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4692118}
I20250623 08:04:46.969659 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 434) took 4978 ms. Trace:
I20250623 08:04:46.969727 981 rpcz_store.cc:276] 0623 08:04:41.990924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.990971 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:46.969524 (+4978553us) ts_tablet_manager.cc:1145] Deleting tablet d6ed4cb252b1452087f8ee53ec78d985
0623 08:04:46.969532 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.969645 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4856340}
I20250623 08:04:46.969945 981 tablet_replica.cc:331] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:46.970098 981 raft_consensus.cc:2241] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:46.970278 981 pending_rounds.cc:70] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Trying to abort 1 pending ops.
I20250623 08:04:46.970312 981 pending_rounds.cc:77] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7170727472049676288 op_type: NO_OP noop_request { }
I20250623 08:04:46.970535 981 raft_consensus.cc:2887] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:46.970624 981 raft_consensus.cc:2270] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:46.974865 32359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486f00b38c444ea9ba6f6482e05f873d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.978650 32357 tablet_service.cc:1468] Processing CreateTablet for tablet cd457e5206a448d084a616cda7f410c3 (DEFAULT_TABLE table=default.Table_992eb98f4db0454083b215b3ac3d38b7 [id=743dc03385c4498c8c1ca8df7e859bee]), partition=RANGE (key) PARTITION UNBOUNDED
W20250623 08:04:46.979388 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 425) could send its response
W20250623 08:04:46.979540 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 434) could send its response
I20250623 08:04:46.919672 339 tablet_bootstrap.cc:492] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:46.920930 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 51fce46407ac4e57a74ad696e85daac7 (DEFAULT_TABLE table=default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.979763 339 ts_tablet_manager.cc:1397] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.000s
I20250623 08:04:46.979876 2455 ts_tablet_manager.cc:1918] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:46.979918 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51fce46407ac4e57a74ad696e85daac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.979974 2455 log.cc:1199] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/bb06a340f82047ac8ec3516edb08f4b6
I20250623 08:04:46.932286 32348 tablet_service.cc:1468] Processing CreateTablet for tablet fa41bab84b384c10ace3ce176698ebe4 (DEFAULT_TABLE table=default.Table_cace1e64be664c9c81b588e80d704aaa [id=7ed25123fbc749909d1d90134be64278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.980304 2455 ts_tablet_manager.cc:1939] T bb06a340f82047ac8ec3516edb08f4b6 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:46.940330 32342 tablet_service.cc:1468] Processing CreateTablet for tablet 5930085e18d84823b27791a65750663f (DEFAULT_TABLE table=default.Table_1631659a168049fba9411207edd8b56f [id=10d3dacba1db466893ef47b1c42a8f15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.948885 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 83e8e9739af0464787afc3580a034cd6 (DEFAULT_TABLE table=default.Table_43ff9ceffb4042b5ab1b041b76bdfc96 [id=2013d4281dc84c089dc17304aefd0837]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.980476 339 raft_consensus.cc:357] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.961737 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 049b9194c9f74b8997c06469d74b9795 (DEFAULT_TABLE table=default.Table_4d2fc00363204f8485905af018f7911a [id=22f134d5d55f40ef98dec2f14a6d55ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.968191 32372 raft_consensus.cc:3058] T b61b02cf2904404f8c0ffef7d268fefc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:46.914649 32354 tablet_service.cc:1468] Processing CreateTablet for tablet e259ef662ee44d1ca5081c05e9fed5f6 (DEFAULT_TABLE table=default.Table_bc37bf6911b5401c904eef106dde3592 [id=19ae0916b8c6480f93b92da98de7ec4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.916481 32350 tablet_service.cc:1468] Processing CreateTablet for tablet ebe6148c11144fdb8519a3973e822159 (DEFAULT_TABLE table=default.Table_306d24f2f3484139b3586e0a1ebb8562 [id=2f72a0d782b54f109937d1ab340afd89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:46.980641 339 raft_consensus.cc:383] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:46.980960 339 raft_consensus.cc:738] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:46.981155 339 consensus_queue.cc:260] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:46.981674 339 ts_tablet_manager.cc:1428] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:46.981926 339 tablet_bootstrap.cc:492] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:46.983799 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebe6148c11144fdb8519a3973e822159. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.984249 32342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5930085e18d84823b27791a65750663f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.984625 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83e8e9739af0464787afc3580a034cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.985026 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 049b9194c9f74b8997c06469d74b9795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.985474 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa41bab84b384c10ace3ce176698ebe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.985880 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e259ef662ee44d1ca5081c05e9fed5f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:46.987604 339 tablet_bootstrap.cc:654] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:46.988710 2242 consensus_queue.cc:1035] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:46.941358 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 402) took 5716 ms. Trace:
I20250623 08:04:46.989583 952 rpcz_store.cc:276] 0623 08:04:41.224832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.224875 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:46.941251 (+5716376us) ts_tablet_manager.cc:1145] Deleting tablet 74c1cdc83ffd4a09a163479ac58fd405
0623 08:04:46.941258 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:46.941346 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5716188}
I20250623 08:04:46.989979 952 tablet_replica.cc:331] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:46.990191 952 raft_consensus.cc:2241] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:46.990372 952 raft_consensus.cc:2270] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:46.991024 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 402) could send its response
I20250623 08:04:46.999361 952 ts_tablet_manager.cc:1905] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.010130 339 tablet_bootstrap.cc:492] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:47.010365 339 ts_tablet_manager.cc:1397] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.003s
I20250623 08:04:47.011210 339 raft_consensus.cc:357] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.011435 339 raft_consensus.cc:383] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.011526 339 raft_consensus.cc:738] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.011759 339 consensus_queue.cc:260] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.013103 339 ts_tablet_manager.cc:1428] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:47.013971 339 tablet_bootstrap.cc:492] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:47.015620 339 tablet_bootstrap.cc:654] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.016597 2242 consensus_queue.cc:1035] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.025283 1957 consensus_queue.cc:1035] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.026114 32381 raft_consensus.cc:3058] T 7dc5e0fbeba4412aa42d116cd8882238 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.026892 2197 server_base.cc:1048] running on GCE node
W20250623 08:04:47.028045 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250623 08:04:47.029094 2197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250623 08:04:47.040148 2197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20250623 08:04:47.041325 32453 consensus_peers.cc:487] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.041667 2197 hybrid_clock.cc:648] HybridClock initialized: now 1750665887041560 us; error 96 us; skew 500 ppm
I20250623 08:04:47.042115 2197 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250623 08:04:47.048110 2446 raft_consensus.cc:491] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.048254 2446 raft_consensus.cc:513] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.048863 2446 leader_election.cc:290] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.049399 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c74e8c697d244ac2ac8174bb1fc76e61" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.049408 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c74e8c697d244ac2ac8174bb1fc76e61" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.050175 32314 leader_election.cc:304] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:47.050436 2446 raft_consensus.cc:2747] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.056687 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd457e5206a448d084a616cda7f410c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.059273 32486 tablet_service.cc:1468] Processing CreateTablet for tablet cd457e5206a448d084a616cda7f410c3 (DEFAULT_TABLE table=default.Table_992eb98f4db0454083b215b3ac3d38b7 [id=743dc03385c4498c8c1ca8df7e859bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.059489 952 ts_tablet_manager.cc:1918] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:47.059597 952 log.cc:1199] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/3136ec837bea4619ae3995ceec9b24d6
I20250623 08:04:47.059748 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd457e5206a448d084a616cda7f410c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.059964 952 ts_tablet_manager.cc:1939] T 3136ec837bea4619ae3995ceec9b24d6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:47.060290 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 404) took 5750 ms. Trace:
I20250623 08:04:47.060375 952 rpcz_store.cc:276] 0623 08:04:41.310081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.310150 (+ 69us) service_pool.cc:225] Handling call
0623 08:04:46.989927 (+5679777us) ts_tablet_manager.cc:1145] Deleting tablet 3136ec837bea4619ae3995ceec9b24d6
0623 08:04:46.989938 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.002453 (+ 12515us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.059370 (+ 56917us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.060276 (+ 906us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5679548}
I20250623 08:04:47.060722 952 tablet_replica.cc:331] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:47.060916 952 raft_consensus.cc:2241] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:47.061223 952 raft_consensus.cc:2270] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.061779 952 ts_tablet_manager.cc:1905] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:47.062448 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 404) could send its response
I20250623 08:04:47.064118 2197 webserver.cc:469] Webserver started at http://127.31.69.62:33855/ using document root <none> and password file <none>
I20250623 08:04:47.066895 2197 fs_manager.cc:362] Metadata directory not provided
I20250623 08:04:47.067317 2197 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250623 08:04:47.067587 2446 raft_consensus.cc:491] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.067757 2446 raft_consensus.cc:513] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.068398 2446 leader_election.cc:290] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.069058 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6857437f85ab40f6a24e74ce3b5efbdc" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.069166 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6857437f85ab40f6a24e74ce3b5efbdc" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.069286 32514 raft_consensus.cc:2391] T 6857437f85ab40f6a24e74ce3b5efbdc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:47.070060 32314 leader_election.cc:304] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:47.070856 2446 raft_consensus.cc:3058] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.073738 2446 raft_consensus.cc:2747] T 6857437f85ab40f6a24e74ce3b5efbdc P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.076171 2242 consensus_queue.cc:1035] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:47.080241 32589 consensus_peers.cc:487] T 4cf1f3943a3d49318c435921ebeaaf33 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.082288 32629 tablet_service.cc:1468] Processing CreateTablet for tablet cd457e5206a448d084a616cda7f410c3 (DEFAULT_TABLE table=default.Table_992eb98f4db0454083b215b3ac3d38b7 [id=743dc03385c4498c8c1ca8df7e859bee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.082798 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd457e5206a448d084a616cda7f410c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.098364 1823 consensus_queue.cc:1035] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.098639 32428 heartbeater.cc:499] Master 127.31.69.61:33313 was elected leader, sending a full tablet report...
W20250623 08:04:47.108970 32589 consensus_peers.cc:487] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.133137 2197 fs_manager.cc:714] Time spent opening directory manager: real 0.032s user 0.001s sys 0.007s
I20250623 08:04:47.133777 2329 consensus_queue.cc:1035] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.144814 342 tablet_bootstrap.cc:492] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:47.144974 342 ts_tablet_manager.cc:1397] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.381s user 0.006s sys 0.001s
I20250623 08:04:47.145782 342 raft_consensus.cc:357] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.146150 342 raft_consensus.cc:383] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.146214 342 raft_consensus.cc:738] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.146489 342 consensus_queue.cc:260] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.148730 342 ts_tablet_manager.cc:1428] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:47.150102 342 tablet_bootstrap.cc:492] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:47.151314 952 ts_tablet_manager.cc:1918] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:47.152961 32375 raft_consensus.cc:3058] T c74e8c697d244ac2ac8174bb1fc76e61 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.159842 2242 raft_consensus.cc:491] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:47.161015 2502 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250623 08:04:47.160205 2242 raft_consensus.cc:513] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.162271 2242 leader_election.cc:290] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.165674 2197 fs_manager.cc:730] Time spent opening block manager: real 0.029s user 0.002s sys 0.001s
W20250623 08:04:47.163569 32589 leader_election.cc:343] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 803d3607d75443568b7ddc9f0c0bdc90
W20250623 08:04:47.167044 32589 leader_election.cc:343] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 803d3607d75443568b7ddc9f0c0bdc90
I20250623 08:04:47.167163 32589 leader_election.cc:304] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:47.167644 2329 raft_consensus.cc:2747] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:47.165932 2197 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data,/tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
uuid: "d31835ec4ad74332a00998d431492466"
format_stamp: "Formatted at 2025-06-23 08:04:18 on dist-test-slave-bxhn"
I20250623 08:04:47.168422 2197 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250623 08:04:47.174108 1235 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 27) took 1294 ms. Trace:
I20250623 08:04:47.174284 1235 rpcz_store.cc:276] 0623 08:04:45.879847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.655323 (+775476us) service_pool.cc:225] Handling call
0623 08:04:46.681258 (+ 25935us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d8313 00007f79d9221c96 00007f79d92219c5 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:46.740526 (+ 59268us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:46.741053 (+ 527us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:47.173752 (+432699us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:47.173920 (+ 168us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:47.174056 (+ 136us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:46.741550 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:46.741669 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:46.741676 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:46.741679 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:46.741821 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:46.741838 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:46.741846 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:46.741954 (+ 108us) tablet.cc:699] Row locks acquired
0623 08:04:46.741959 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:46.741997 (+ 38us) write_op.cc:270] Start()
0623 08:04:46.742050 (+ 53us) write_op.cc:276] Timestamp: P: 1750665886741935 usec, L: 0
0623 08:04:46.742054 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:46.742216 (+ 162us) log.cc:844] Serialized 433 byte log entry
0623 08:04:47.171967 (+429751us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.172157 (+ 190us) write_op.cc:301] APPLY: starting
0623 08:04:47.172179 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.172261 (+ 82us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.172297 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.172444 (+ 147us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.172514 (+ 70us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:47.172527 (+ 13us) write_op.cc:312] APPLY: finished
0623 08:04:47.172633 (+ 106us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.172723 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.172749 (+ 26us) write_op.cc:454] Released schema lock
0623 08:04:47.172759 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.172850 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.172857 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":56596864,"child_traces":[["op",{"apply.queue_time_us":123,"apply.run_cpu_time_us":682,"apply.run_wall_time_us":682,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":753,"prepare.run_wall_time_us":754,"raft.queue_time_us":58,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":429866}]]}
I20250623 08:04:47.179821 1225 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 25) took 1299 ms. Trace:
I20250623 08:04:47.180548 952 log.cc:1199] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/92005abc61dd40ef9da8124cadc29b6a
I20250623 08:04:47.180009 1225 rpcz_store.cc:276] 0623 08:04:45.879561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.572148 (+692587us) service_pool.cc:225] Handling call
0623 08:04:46.689266 (+117118us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d8313 00007f79d9221c96 00007f79d92219c5 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:46.743294 (+ 54028us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:46.743787 (+ 493us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:47.178472 (+434685us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:47.178598 (+ 126us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:47.178718 (+ 120us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:46.744319 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:46.744417 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:46.744423 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:46.744425 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:46.744507 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:46.744518 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:46.744524 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:46.744592 (+ 68us) tablet.cc:699] Row locks acquired
0623 08:04:46.744594 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:46.744625 (+ 31us) write_op.cc:270] Start()
0623 08:04:46.744658 (+ 33us) write_op.cc:276] Timestamp: P: 1750665886744563 usec, L: 0
0623 08:04:46.744662 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:46.744814 (+ 152us) log.cc:844] Serialized 433 byte log entry
0623 08:04:47.172071 (+427257us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.172930 (+ 859us) write_op.cc:301] APPLY: starting
0623 08:04:47.172940 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.172967 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.172995 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.173099 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.173128 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:47.173137 (+ 9us) write_op.cc:312] APPLY: finished
0623 08:04:47.173191 (+ 54us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.173233 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.173250 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:47.173259 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":248590208,"child_traces":[["op",{"apply.queue_time_us":821,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":398,"num_ops":2,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":572,"prepare.run_wall_time_us":573,"raft.queue_time_us":55,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":427387}]]}
I20250623 08:04:47.180953 952 ts_tablet_manager.cc:1939] T 92005abc61dd40ef9da8124cadc29b6a P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:47.181308 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 408) took 5257 ms. Trace:
I20250623 08:04:47.181399 952 rpcz_store.cc:276] 0623 08:04:41.923657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.923829 (+ 172us) service_pool.cc:225] Handling call
0623 08:04:47.060681 (+5136852us) ts_tablet_manager.cc:1145] Deleting tablet 92005abc61dd40ef9da8124cadc29b6a
0623 08:04:47.060690 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.105906 (+ 45216us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.151198 (+ 45292us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.181287 (+ 30089us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5135746}
W20250623 08:04:47.181653 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 408) could send its response
I20250623 08:04:47.184917 1225 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_1631659a168049fba9411207edd8b56f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:47.186676 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 409) took 5262 ms. Trace:
I20250623 08:04:47.186817 952 rpcz_store.cc:276] 0623 08:04:41.924059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.924232 (+ 173us) service_pool.cc:225] Handling call
0623 08:04:47.186428 (+5262196us) ts_tablet_manager.cc:1145] Deleting tablet cdec6e9573b041e0a57d1e0e0239d97f
0623 08:04:47.186441 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.186656 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5260759}
W20250623 08:04:47.187067 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 409) could send its response
I20250623 08:04:47.187399 952 tablet_replica.cc:331] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:47.187634 952 raft_consensus.cc:2241] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:47.188483 952 raft_consensus.cc:2270] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.189095 952 ts_tablet_manager.cc:1905] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.190193 2446 raft_consensus.cc:491] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.190340 2446 raft_consensus.cc:513] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.190987 2446 leader_election.cc:290] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.194841 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea588a48e6164d3db148fdaa306268f6" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.195050 32514 raft_consensus.cc:2391] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
W20250623 08:04:47.196122 32453 consensus_peers.cc:487] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d19d468ce13d4108a9badc36fc503306. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.196390 1248 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_b68f23253200457683bb5cedcc56041f" } new_table_name: "default.Table_682a42dbf21d43d49ad9d85b958b121c" modify_external_catalogs: true
W20250623 08:04:47.197667 1225 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1631659a168049fba9411207edd8b56f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:47.198010 1225 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 33) took 1317 ms. Trace:
I20250623 08:04:47.198130 1225 rpcz_store.cc:276] 0623 08:04:45.880466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.180739 (+1300273us) service_pool.cc:225] Handling call
0623 08:04:47.184861 (+ 4122us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d8313 00007f79d9221c96 00007f79d92219c5 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:47.197806 (+ 12945us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:47.197975 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8800128}
I20250623 08:04:47.198455 1225 master_service.cc:948] ReplaceTablet: received request to replace tablet a2a07188a4d340bc9caf010d5202e34d from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:47.198905 1225 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 34) took 1318 ms. Trace:
I20250623 08:04:47.199002 1225 rpcz_store.cc:276] 0623 08:04:45.880544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.198393 (+1317849us) service_pool.cc:225] Handling call
0623 08:04:47.198851 (+ 458us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 08:04:47.199250 1225 master_service.cc:948] ReplaceTablet: received request to replace tablet 9fd478c007914d2cb09d929b50322a64 from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:47.199577 1225 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 35) took 1318 ms. Trace:
I20250623 08:04:47.199688 1225 rpcz_store.cc:276] 0623 08:04:45.880619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.199186 (+1318567us) service_pool.cc:225] Handling call
0623 08:04:47.199546 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250623 08:04:47.199982 1225 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_217651efa5dc4571b2a47cd3196c9e57" } new_table_name: "default.Table_8f362905029746c19e0e7ce1433c594d" modify_external_catalogs: true
I20250623 08:04:47.200737 952 ts_tablet_manager.cc:1918] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:47.201013 952 log.cc:1199] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/4c9d1c8a67e149478a0d69842a4eb5b2
I20250623 08:04:47.201447 952 ts_tablet_manager.cc:1939] T 4c9d1c8a67e149478a0d69842a4eb5b2 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:47.201817 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 411) took 5277 ms. Trace:
I20250623 08:04:47.201932 952 rpcz_store.cc:276] 0623 08:04:41.924617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.924774 (+ 157us) service_pool.cc:225] Handling call
0623 08:04:47.187285 (+5262511us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d1c8a67e149478a0d69842a4eb5b2
0623 08:04:47.187295 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.196310 (+ 9015us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.200634 (+ 4324us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.201802 (+ 1168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5262274}
I20250623 08:04:47.202280 952 tablet_replica.cc:331] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:47.202519 952 raft_consensus.cc:2241] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:47.202713 952 raft_consensus.cc:2270] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.202816 1226 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_49e1ee7e0d024625988be24a823a443c" } new_table_name: "default.Table_e4c31249050c4016891036cf9507b607" modify_external_catalogs: true
W20250623 08:04:47.202237 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 411) could send its response
I20250623 08:04:47.204545 952 ts_tablet_manager.cc:1905] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.207628 981 ts_tablet_manager.cc:1905] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:47.210790 1213 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 100983us
I20250623 08:04:47.215183 343 tablet_bootstrap.cc:492] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:47.215341 343 ts_tablet_manager.cc:1397] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.348s user 0.014s sys 0.010s
I20250623 08:04:47.216051 343 raft_consensus.cc:357] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.216207 343 raft_consensus.cc:383] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.216248 343 raft_consensus.cc:738] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.216420 343 consensus_queue.cc:260] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.216898 343 ts_tablet_manager.cc:1428] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250623 08:04:47.219208 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:47.220187 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 540314ce8e3a44c1afb43a1e60b07ca8 (DEFAULT_TABLE table=default.Table_47c06fdd0628467d8a0727b9adeb890d [id=4aea4a71b2e64e81bf9cca2a06f15521]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.220573 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 540314ce8e3a44c1afb43a1e60b07ca8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:47.236697 32589 consensus_peers.cc:487] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:47.242486 32314 leader_election.cc:343] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: ea588a48e6164d3db148fdaa306268f6
I20250623 08:04:47.242630 32314 leader_election.cc:304] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:47.243049 2446 raft_consensus.cc:3058] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.245645 2446 raft_consensus.cc:2747] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.246019 32345 tablet_service.cc:1468] Processing CreateTablet for tablet b91d3650e56441189c3cb74b01cd7fa7 (DEFAULT_TABLE table=default.Table_1ced320c61ec4355bb2686a5c84a55eb [id=61c080721215406397c00aff27538c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.246403 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b91d3650e56441189c3cb74b01cd7fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.247364 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 2ec871b2878a4b3f929b3774746d7750 (DEFAULT_TABLE table=default.Table_787415ae5937475f90f90000aa575d0e [id=4b27489168854629848531915104847a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.247727 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ec871b2878a4b3f929b3774746d7750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.251005 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 2f7ba3b78aab4871a8f07e4ecc24acd4 (DEFAULT_TABLE table=default.Table_100f074733ca4066983e68257aea0b11 [id=5d78d651cb7345fba13c371f9c1db55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.251318 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7ba3b78aab4871a8f07e4ecc24acd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.252631 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 540314ce8e3a44c1afb43a1e60b07ca8 (DEFAULT_TABLE table=default.Table_47c06fdd0628467d8a0727b9adeb890d [id=4aea4a71b2e64e81bf9cca2a06f15521]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.252929 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 540314ce8e3a44c1afb43a1e60b07ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.255088 343 tablet_bootstrap.cc:492] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:47.255800 339 tablet_bootstrap.cc:492] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:47.255793 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 2ec871b2878a4b3f929b3774746d7750 (DEFAULT_TABLE table=default.Table_787415ae5937475f90f90000aa575d0e [id=4b27489168854629848531915104847a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.256224 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ec871b2878a4b3f929b3774746d7750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.258591 1823 consensus_queue.cc:1035] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.258637 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 2f7ba3b78aab4871a8f07e4ecc24acd4 (DEFAULT_TABLE table=default.Table_100f074733ca4066983e68257aea0b11 [id=5d78d651cb7345fba13c371f9c1db55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.259043 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7ba3b78aab4871a8f07e4ecc24acd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.260159 339 ts_tablet_manager.cc:1397] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.247s user 0.035s sys 0.024s
I20250623 08:04:47.261132 339 raft_consensus.cc:357] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.261395 339 raft_consensus.cc:383] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.261507 339 raft_consensus.cc:738] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.261803 339 consensus_queue.cc:260] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.262511 339 ts_tablet_manager.cc:1428] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:47.263135 32499 tablet_service.cc:1468] Processing CreateTablet for tablet 2f7ba3b78aab4871a8f07e4ecc24acd4 (DEFAULT_TABLE table=default.Table_100f074733ca4066983e68257aea0b11 [id=5d78d651cb7345fba13c371f9c1db55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.263585 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7ba3b78aab4871a8f07e4ecc24acd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.263880 339 tablet_bootstrap.cc:492] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:47.265491 339 tablet_bootstrap.cc:654] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.283795 339 tablet_bootstrap.cc:492] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:47.283866 32630 tablet_service.cc:1468] Processing CreateTablet for tablet b91d3650e56441189c3cb74b01cd7fa7 (DEFAULT_TABLE table=default.Table_1ced320c61ec4355bb2686a5c84a55eb [id=61c080721215406397c00aff27538c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.284024 339 ts_tablet_manager.cc:1397] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250623 08:04:47.284322 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b91d3650e56441189c3cb74b01cd7fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.284906 339 raft_consensus.cc:357] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.285140 339 raft_consensus.cc:383] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.285226 339 raft_consensus.cc:738] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.285446 339 consensus_queue.cc:260] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.285986 339 ts_tablet_manager.cc:1428] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:47.290659 339 tablet_bootstrap.cc:492] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:47.291128 343 tablet_bootstrap.cc:654] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.292680 339 tablet_bootstrap.cc:654] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.294448 32348 tablet_service.cc:1468] Processing CreateTablet for tablet b13dfc9afcbf43c685ca49d5ae0c5560 (DEFAULT_TABLE table=default.Table_9a49c84f75b84594ab81fcf31da3991f [id=7b34109f27f5441e855b23fb253a8e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.294857 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13dfc9afcbf43c685ca49d5ae0c5560. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.307237 32626 tablet_service.cc:1468] Processing CreateTablet for tablet b13dfc9afcbf43c685ca49d5ae0c5560 (DEFAULT_TABLE table=default.Table_9a49c84f75b84594ab81fcf31da3991f [id=7b34109f27f5441e855b23fb253a8e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.265762 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 2ec871b2878a4b3f929b3774746d7750 (DEFAULT_TABLE table=default.Table_787415ae5937475f90f90000aa575d0e [id=4b27489168854629848531915104847a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.307950 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ec871b2878a4b3f929b3774746d7750. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.309300 2329 raft_consensus.cc:491] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.309422 2329 raft_consensus.cc:513] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.310058 2329 leader_election.cc:290] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.310139 32345 tablet_service.cc:1468] Processing CreateTablet for tablet a36c01f33f4b423aba86bd9a7f212c5d (DEFAULT_TABLE table=default.Table_87e0e2fcec324a53ae2cfd5b3156e071 [id=75e5879d4117409da29206bb682898cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.310676 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a36c01f33f4b423aba86bd9a7f212c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.310962 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c3dcdcaa2e14cb2bd02ebb8f1d0e159" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:47.311131 32372 raft_consensus.cc:2466] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:47.311568 32589 leader_election.cc:304] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:47.311838 2337 raft_consensus.cc:2802] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:47.311906 2337 raft_consensus.cc:491] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:47.311960 2337 raft_consensus.cc:3058] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Advancing to term 2
W20250623 08:04:47.313467 32589 leader_election.cc:343] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 6c3dcdcaa2e14cb2bd02ebb8f1d0e159
I20250623 08:04:47.314049 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13dfc9afcbf43c685ca49d5ae0c5560. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.314503 32629 tablet_service.cc:1468] Processing CreateTablet for tablet a36c01f33f4b423aba86bd9a7f212c5d (DEFAULT_TABLE table=default.Table_87e0e2fcec324a53ae2cfd5b3156e071 [id=75e5879d4117409da29206bb682898cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.314906 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a36c01f33f4b423aba86bd9a7f212c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.321378 1953 consensus_queue.cc:1035] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.264762 342 tablet_bootstrap.cc:654] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.324597 2337 raft_consensus.cc:513] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.269356 32483 tablet_service.cc:1468] Processing CreateTablet for tablet b91d3650e56441189c3cb74b01cd7fa7 (DEFAULT_TABLE table=default.Table_1ced320c61ec4355bb2686a5c84a55eb [id=61c080721215406397c00aff27538c61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.328447 981 ts_tablet_manager.cc:1918] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250623 08:04:47.269528 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 540314ce8e3a44c1afb43a1e60b07ca8 (DEFAULT_TABLE table=default.Table_47c06fdd0628467d8a0727b9adeb890d [id=4aea4a71b2e64e81bf9cca2a06f15521]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.293426 32489 tablet_service.cc:1468] Processing CreateTablet for tablet a36c01f33f4b423aba86bd9a7f212c5d (DEFAULT_TABLE table=default.Table_87e0e2fcec324a53ae2cfd5b3156e071 [id=75e5879d4117409da29206bb682898cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.293642 32484 tablet_service.cc:1468] Processing CreateTablet for tablet b13dfc9afcbf43c685ca49d5ae0c5560 (DEFAULT_TABLE table=default.Table_9a49c84f75b84594ab81fcf31da3991f [id=7b34109f27f5441e855b23fb253a8e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.331333 2197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250623 08:04:47.333170 32483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b91d3650e56441189c3cb74b01cd7fa7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:47.338310 1229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:54519 (request call id 139) took 1299 ms (client timeout 1000 ms). Trace:
W20250623 08:04:47.338456 1229 rpcz_store.cc:269] 0623 08:04:46.038477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.515893 (+477416us) service_pool.cc:225] Handling call
0623 08:04:46.668740 (+152847us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d2b58 00007f79d921f176 00007f79d921eea5 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:47.338260 (+669520us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:46.820270 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:46.820410 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:46.820420 (+ 10us) write_op.cc:435] Acquired schema lock
0623 08:04:46.820422 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:46.820540 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:46.820552 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:46.820557 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:46.820639 (+ 82us) tablet.cc:699] Row locks acquired
0623 08:04:46.820642 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:46.820678 (+ 36us) write_op.cc:270] Start()
0623 08:04:46.820711 (+ 33us) write_op.cc:276] Timestamp: P: 1750665886820608 usec, L: 0
0623 08:04:46.820714 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.186178 (+365464us) log.cc:844] Serialized 527 byte log entry
0623 08:04:47.186477 (+ 299us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 000000000097041a 00007f79da6a15e2 00007f79dba48255 00007f79dba46d79 00007f79dba4f1a0 00007f79dba4efad 0000000000a751bc 00007f79cecb33ff 00007f79cecb66cb 00007f79cecb6548 0000000000a751bc
0623 08:04:47.335924 (+149447us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.336113 (+ 189us) write_op.cc:301] APPLY: starting
0623 08:04:47.336131 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.336176 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.336179 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.336234 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.336275 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.336283 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:47.336367 (+ 84us) log.cc:844] Serialized 23 byte log entry
0623 08:04:47.336437 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.336456 (+ 19us) write_op.cc:454] Released schema lock
0623 08:04:47.336462 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.336532 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.336536 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":347089792,"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":422,"apply.run_wall_time_us":422,"num_ops":1,"prepare.queue_time_us":66,"prepare.run_cpu_time_us":3661,"prepare.run_wall_time_us":366276,"raft.queue_time_us":1948,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":317,"replication_time_us":515164,"spinlock_wait_cycles":839985024}]]}
I20250623 08:04:47.339130 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a36c01f33f4b423aba86bd9a7f212c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.343405 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13dfc9afcbf43c685ca49d5ae0c5560. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.350170 2197 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250623 08:04:47.353745 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet adc3cf4355c24b7395fcb22af8520346 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:47.354152 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 540314ce8e3a44c1afb43a1e60b07ca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.359524 342 tablet_bootstrap.cc:492] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:47.359704 342 ts_tablet_manager.cc:1397] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.210s user 0.003s sys 0.000s
I20250623 08:04:47.360477 342 raft_consensus.cc:357] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.360672 342 raft_consensus.cc:383] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.360754 342 raft_consensus.cc:738] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.360952 342 consensus_queue.cc:260] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.364567 342 ts_tablet_manager.cc:1428] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:47.367142 2337 leader_election.cc:290] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.368597 2197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250623 08:04:47.375842 981 log.cc:1199] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/a8de2b1105564fdab386442b1de4cedb
I20250623 08:04:47.376376 981 ts_tablet_manager.cc:1939] T a8de2b1105564fdab386442b1de4cedb P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:47.382663 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 447) took 4829 ms. Trace:
I20250623 08:04:47.375720 339 tablet_bootstrap.cc:492] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:47.385991 339 ts_tablet_manager.cc:1397] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.007s
I20250623 08:04:47.386987 339 raft_consensus.cc:357] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.387233 339 raft_consensus.cc:383] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.387437 339 raft_consensus.cc:738] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.387710 339 consensus_queue.cc:260] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.385064 981 rpcz_store.cc:276] 0623 08:04:42.553430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.553492 (+ 62us) service_pool.cc:225] Handling call
0623 08:04:46.969909 (+4416417us) ts_tablet_manager.cc:1145] Deleting tablet a8de2b1105564fdab386442b1de4cedb
0623 08:04:46.969917 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.324805 (+354888us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.328343 (+ 3538us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.382630 (+ 54287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4416160}
W20250623 08:04:47.391680 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 447) could send its response
I20250623 08:04:47.391746 981 tablet_replica.cc:331] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:47.392102 339 ts_tablet_manager.cc:1428] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:47.392622 339 tablet_bootstrap.cc:492] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:47.392302 981 raft_consensus.cc:2241] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:47.393998 981 raft_consensus.cc:2270] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.395027 981 ts_tablet_manager.cc:1905] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.395936 343 tablet_bootstrap.cc:492] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:47.396126 343 ts_tablet_manager.cc:1397] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.141s user 0.004s sys 0.005s
I20250623 08:04:47.397020 343 raft_consensus.cc:357] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.397264 343 raft_consensus.cc:383] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.397359 343 raft_consensus.cc:738] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.397639 343 consensus_queue.cc:260] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.398286 343 ts_tablet_manager.cc:1428] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:47.398669 343 tablet_bootstrap.cc:492] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:47.400313 343 tablet_bootstrap.cc:654] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.410259 339 tablet_bootstrap.cc:654] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.418715 2446 raft_consensus.cc:491] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.418900 2446 raft_consensus.cc:513] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.419862 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de743139c4324faba19ff3e5cfac855d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.419462 2446 leader_election.cc:290] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.420361 981 ts_tablet_manager.cc:1918] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:47.420557 981 log.cc:1199] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/3740dd120745493688eeb4c1ac311dcb
I20250623 08:04:47.420785 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de743139c4324faba19ff3e5cfac855d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.420915 981 ts_tablet_manager.cc:1939] T 3740dd120745493688eeb4c1ac311dcb P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:47.421331 32314 leader_election.cc:304] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:47.421653 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 448) took 4868 ms. Trace:
I20250623 08:04:47.421746 981 rpcz_store.cc:276] 0623 08:04:42.553551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.553608 (+ 57us) service_pool.cc:225] Handling call
0623 08:04:47.391697 (+4838089us) ts_tablet_manager.cc:1145] Deleting tablet 3740dd120745493688eeb4c1ac311dcb
0623 08:04:47.391707 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.415501 (+ 23794us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.420261 (+ 4760us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.421631 (+ 1370us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":323,"tablet-delete.queue_time_us":4837881}
W20250623 08:04:47.421998 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 448) could send its response
I20250623 08:04:47.422268 343 tablet_bootstrap.cc:492] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:47.422570 343 ts_tablet_manager.cc:1397] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250623 08:04:47.423599 981 tablet_replica.cc:331] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:47.423807 981 raft_consensus.cc:2241] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:47.424598 981 raft_consensus.cc:2270] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.425179 981 ts_tablet_manager.cc:1905] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.426391 2446 raft_consensus.cc:2747] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.427054 343 raft_consensus.cc:357] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.427307 343 raft_consensus.cc:383] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.427413 343 raft_consensus.cc:738] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.427893 343 consensus_queue.cc:260] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.428470 343 ts_tablet_manager.cc:1428] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250623 08:04:47.428786 343 tablet_bootstrap.cc:492] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:47.430608 343 tablet_bootstrap.cc:654] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.438711 981 ts_tablet_manager.cc:1918] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:47.440326 981 log.cc:1199] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/a9ea74d20c5f4eefbcb8508ed1774026
I20250623 08:04:47.440855 981 ts_tablet_manager.cc:1939] T a9ea74d20c5f4eefbcb8508ed1774026 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:47.442097 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 449) took 4888 ms. Trace:
W20250623 08:04:47.443917 32453 consensus_peers.cc:487] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.443084 981 rpcz_store.cc:276] 0623 08:04:42.554040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.554865 (+ 825us) service_pool.cc:225] Handling call
0623 08:04:47.423539 (+4868674us) ts_tablet_manager.cc:1145] Deleting tablet a9ea74d20c5f4eefbcb8508ed1774026
0623 08:04:47.423549 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.430613 (+ 7064us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.438602 (+ 7989us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.442077 (+ 3475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4868526}
W20250623 08:04:47.444548 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 449) could send its response
I20250623 08:04:47.444885 981 tablet_replica.cc:331] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:47.445293 981 raft_consensus.cc:2241] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:47.445354 343 tablet_bootstrap.cc:492] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:47.445560 343 ts_tablet_manager.cc:1397] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20250623 08:04:47.445729 981 raft_consensus.cc:2270] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.446494 343 raft_consensus.cc:357] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.446707 343 raft_consensus.cc:383] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.446866 343 raft_consensus.cc:738] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.446475 981 ts_tablet_manager.cc:1905] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.447307 343 consensus_queue.cc:260] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.447912 343 ts_tablet_manager.cc:1428] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:47.448271 343 tablet_bootstrap.cc:492] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:47.450563 343 tablet_bootstrap.cc:654] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.451365 2446 raft_consensus.cc:491] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.451591 2446 raft_consensus.cc:513] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.452214 2446 leader_election.cc:290] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.452934 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f957f5575e9475eb9f639e3ad5cb809" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.453502 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f957f5575e9475eb9f639e3ad5cb809" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.454031 32314 leader_election.cc:304] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:47.454284 2446 raft_consensus.cc:2747] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.455739 342 tablet_bootstrap.cc:492] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:47.457250 342 tablet_bootstrap.cc:654] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.464172 981 ts_tablet_manager.cc:1918] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250623 08:04:47.465157 32453 consensus_peers.cc:487] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.465925 981 log.cc:1199] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/1be8f8557b9a4ae2917c37b8d01cedf6
I20250623 08:04:47.466907 981 ts_tablet_manager.cc:1939] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:47.467837 981 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 446) took 4914 ms. Trace:
I20250623 08:04:47.468792 981 rpcz_store.cc:276] 0623 08:04:42.553302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.571398 (+ 18096us) service_pool.cc:225] Handling call
0623 08:04:47.444838 (+4873440us) ts_tablet_manager.cc:1145] Deleting tablet 1be8f8557b9a4ae2917c37b8d01cedf6
0623 08:04:47.444848 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.455509 (+ 10661us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.464053 (+ 8544us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.467819 (+ 3766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4873158}
W20250623 08:04:47.469802 32314 connection.cc:441] server connection from 127.0.0.1:35542 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35542 (request call id 446) could send its response
W20250623 08:04:47.470810 1213 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet dcfe68d4e6ae470d816ac4c494dcaa51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcfe68d4e6ae470d816ac4c494dcaa51 already in progress: creating tablet
I20250623 08:04:47.471541 981 tablet_replica.cc:331] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:47.471940 981 raft_consensus.cc:2241] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:47.472669 981 raft_consensus.cc:2270] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.473022 342 tablet_bootstrap.cc:492] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:47.473222 342 ts_tablet_manager.cc:1397] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20250623 08:04:47.473876 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6c3dcdcaa2e14cb2bd02ebb8f1d0e159" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:47.474292 342 raft_consensus.cc:357] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.474443 342 raft_consensus.cc:383] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.474449 32377 raft_consensus.cc:3058] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:47.474498 342 raft_consensus.cc:738] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.474706 342 consensus_queue.cc:260] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.475344 342 ts_tablet_manager.cc:1428] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:47.475716 342 tablet_bootstrap.cc:492] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:47.477047 32377 raft_consensus.cc:2466] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 2.
I20250623 08:04:47.477265 342 tablet_bootstrap.cc:654] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.473400 981 ts_tablet_manager.cc:1905] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.488574 981 ts_tablet_manager.cc:1918] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:47.490010 981 log.cc:1199] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/ea588a48e6164d3db148fdaa306268f6
I20250623 08:04:47.490502 981 ts_tablet_manager.cc:1939] T ea588a48e6164d3db148fdaa306268f6 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:47.490900 342 tablet_bootstrap.cc:492] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:47.492400 342 ts_tablet_manager.cc:1397] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20250623 08:04:47.493117 342 raft_consensus.cc:357] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.493537 1213 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet ea588a48e6164d3db148fdaa306268f6 (table default.Table_b523bec7182443139e891357d295e3a1 [id=1c4298a5f9014d8eb4ccad8c8fbdeeac]) successfully deleted
I20250623 08:04:47.493608 342 raft_consensus.cc:383] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.493675 342 raft_consensus.cc:738] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.493903 342 consensus_queue.cc:260] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.494510 342 ts_tablet_manager.cc:1428] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:47.495944 981 tablet_replica.cc:331] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:47.496577 981 raft_consensus.cc:2241] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:47.496903 981 raft_consensus.cc:2270] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.499933 342 tablet_bootstrap.cc:492] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:47.502239 2329 raft_consensus.cc:491] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.502403 2329 raft_consensus.cc:513] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.503281 2329 leader_election.cc:290] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.506422 952 ts_tablet_manager.cc:1918] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:47.507956 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:47.508648 32589 leader_election.cc:304] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:47.508515 1230 catalog_manager.cc:5582] T 306c4021b5d94a4996de07c222a8bfd8 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } }
I20250623 08:04:47.509559 1230 catalog_manager.cc:5582] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.509920 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:47.510205 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:47.510514 1230 catalog_manager.cc:5582] T 3c14e325fa5f4f848748c346bc25dde0 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.511169 1230 catalog_manager.cc:5582] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } }
I20250623 08:04:47.511492 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:47.511798 2242 raft_consensus.cc:2802] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:47.511790 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:47.512064 2242 raft_consensus.cc:695] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [term 2 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:47.511987 1230 catalog_manager.cc:5582] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } }
I20250623 08:04:47.512382 2242 consensus_queue.cc:237] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.512804 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:47.516119 981 ts_tablet_manager.cc:1905] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.516058 2446 raft_consensus.cc:491] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.516351 2446 raft_consensus.cc:513] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.516907 2446 leader_election.cc:290] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.517300 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55669db7e55042b2bc208ea3218c2d6c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.517333 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55669db7e55042b2bc208ea3218c2d6c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.518877 342 tablet_bootstrap.cc:654] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.529651 952 log.cc:1199] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9049e3d9b39f420bb51221808409fc54
I20250623 08:04:47.529701 32314 leader_election.cc:304] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:47.530095 952 ts_tablet_manager.cc:1939] T 9049e3d9b39f420bb51221808409fc54 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:47.530397 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 412) took 5605 ms. Trace:
I20250623 08:04:47.530531 952 rpcz_store.cc:276] 0623 08:04:41.924770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.925255 (+ 485us) service_pool.cc:225] Handling call
0623 08:04:47.202234 (+5276979us) ts_tablet_manager.cc:1145] Deleting tablet 9049e3d9b39f420bb51221808409fc54
0623 08:04:47.202243 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.329855 (+127612us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.506276 (+176421us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.530384 (+ 24108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36,"tablet-delete.queue_time_us":5276817}
W20250623 08:04:47.530900 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 412) could send its response
I20250623 08:04:47.519011 1957 consensus_queue.cc:1035] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.524080 1359 raft_consensus.cc:491] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.531168 1359 raft_consensus.cc:513] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.531927 1359 leader_election.cc:290] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:47.532817 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cca63991d5b4ef5bd8a9181d127ebe3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:47.533334 32453 leader_election.cc:343] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.533632 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cca63991d5b4ef5bd8a9181d127ebe3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.533865 32646 raft_consensus.cc:2466] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:47.534250 32453 leader_election.cc:304] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:47.534722 1359 raft_consensus.cc:491] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.534844 2446 raft_consensus.cc:2747] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.534878 1359 raft_consensus.cc:513] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.535355 1359 leader_election.cc:290] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:47.534637 1957 raft_consensus.cc:2802] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:47.535647 1957 raft_consensus.cc:491] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:47.535779 1957 raft_consensus.cc:3058] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.537585 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74c1cdc83ffd4a09a163479ac58fd405" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:47.538741 32453 leader_election.cc:343] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.540268 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74c1cdc83ffd4a09a163479ac58fd405" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.540505 32647 raft_consensus.cc:2466] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:47.540935 32453 leader_election.cc:304] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:47.541311 1359 raft_consensus.cc:2802] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:47.541409 1359 raft_consensus.cc:491] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:47.541540 1359 raft_consensus.cc:3058] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.545684 1359 raft_consensus.cc:513] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.546280 1359 leader_election.cc:290] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:47.547066 1230 catalog_manager.cc:5582] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.548049 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:47.548161 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74c1cdc83ffd4a09a163479ac58fd405" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:47.548645 1230 catalog_manager.cc:5582] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } }
I20250623 08:04:47.549477 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (table default.Table_4b49e65313cd47e5b682a79f0394d475 [id=ad7210d2665f463c8591e79cf62d15dc]). Expected version 2 got 0
I20250623 08:04:47.549935 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
W20250623 08:04:47.549933 32453 leader_election.cc:343] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.550549 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:47.551184 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:47.551508 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74c1cdc83ffd4a09a163479ac58fd405" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:47.551719 32647 raft_consensus.cc:3058] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.552295 342 tablet_bootstrap.cc:492] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:47.552497 342 ts_tablet_manager.cc:1397] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.007s
I20250623 08:04:47.553289 342 raft_consensus.cc:357] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.553483 342 raft_consensus.cc:383] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.553586 342 raft_consensus.cc:738] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.553900 342 consensus_queue.cc:260] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.554538 342 ts_tablet_manager.cc:1428] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:47.554811 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:47.555029 342 tablet_bootstrap.cc:492] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:47.555011 32647 raft_consensus.cc:2466] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:47.555162 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:47.555611 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:47.556062 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:47.556885 342 tablet_bootstrap.cc:654] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.556857 32453 leader_election.cc:304] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:47.557652 1823 raft_consensus.cc:2802] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:47.557766 1823 raft_consensus.cc:695] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:47.557972 1823 consensus_queue.cc:237] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.559201 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:47.559909 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:47.560873 342 tablet_bootstrap.cc:492] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:47.561030 342 ts_tablet_manager.cc:1397] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250623 08:04:47.560954 1230 catalog_manager.cc:5582] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.561756 342 raft_consensus.cc:357] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.561902 342 raft_consensus.cc:383] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.561980 342 raft_consensus.cc:738] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.562183 342 consensus_queue.cc:260] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.562475 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:47.562961 342 ts_tablet_manager.cc:1428] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:47.563683 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1cee3de5953244f98428747bea912f77 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]). Expected version 1 got 0
I20250623 08:04:47.564160 342 tablet_bootstrap.cc:492] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:47.565392 981 ts_tablet_manager.cc:1918] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:47.567116 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_217651efa5dc4571b2a47cd3196c9e57 [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 1 got 0
I20250623 08:04:47.567682 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:47.568233 1230 catalog_manager.cc:5582] T 5443f12974bf41929d9cd937f146cd56 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.568507 1957 raft_consensus.cc:513] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.568884 981 log.cc:1199] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/bb06a340f82047ac8ec3516edb08f4b6
I20250623 08:04:47.569481 981 ts_tablet_manager.cc:1939] T bb06a340f82047ac8ec3516edb08f4b6 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:47.570235 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1c529e826fa84a919c018887620e4023 (table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]). Expected version 1 got 0
I20250623 08:04:47.570494 1957 leader_election.cc:290] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:47.571203 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cca63991d5b4ef5bd8a9181d127ebe3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:47.571420 32646 raft_consensus.cc:3058] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.570878 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0cca63991d5b4ef5bd8a9181d127ebe3" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:47.572098 32453 leader_election.cc:343] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.573213 1213 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
I20250623 08:04:47.573632 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:47.574643 32646 raft_consensus.cc:2466] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:47.575192 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c50655c06cd94c67a49513c39728eaa9 (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:47.575139 32453 leader_election.cc:304] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:47.576037 1953 raft_consensus.cc:2802] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:47.576167 1953 raft_consensus.cc:695] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:47.576025 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:47.576485 1953 consensus_queue.cc:237] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.579023 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:47.579823 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:47.581079 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:47.582160 1230 catalog_manager.cc:5582] T d6fdea59f71e4a9d9fca8ede4b9439d2 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.583359 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7f890ee0747b4df0b82bc3b396191124 (table default.Table_7ccb5edd6b824f308d349c51d926d963 [id=9f97f6deca194137a353ea221d0e3676]). Expected version 1 got 0
I20250623 08:04:47.586717 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 035bb06bc7dd42f681886def9d56eb73 (table default.Table_53879d6f821f4167bf02ffbe967f73f7 [id=b3f407a7a26e4bd5ae9a719a5c4577c0]). Expected version 1 got 0
I20250623 08:04:47.587424 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:47.587590 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "492ffaeaaa314dcfb7512d8f23946a67" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:47.587981 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:47.588608 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "492ffaeaaa314dcfb7512d8f23946a67" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.589334 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:47.590134 1230 catalog_manager.cc:5582] T affa3da7540648d28c5f00a3cf348eb1 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.591853 1230 catalog_manager.cc:5582] T 45293db3c40b4e4194445f12e7475e66 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } }
W20250623 08:04:47.595865 32453 consensus_peers.cc:487] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.595881 1230 catalog_manager.cc:5582] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.596776 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:47.597321 1230 catalog_manager.cc:5582] T a0e9b2f810014e81a29ca9b41cd3f8ba P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.599038 339 tablet_bootstrap.cc:492] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:47.599205 339 ts_tablet_manager.cc:1397] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.207s user 0.015s sys 0.033s
I20250623 08:04:47.599992 339 raft_consensus.cc:357] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.600195 339 raft_consensus.cc:383] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.600275 339 raft_consensus.cc:738] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.600515 339 consensus_queue.cc:260] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.603068 2517 consensus_queue.cc:1035] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.604730 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 413) took 5679 ms. Trace:
I20250623 08:04:47.604853 952 rpcz_store.cc:276] 0623 08:04:41.925536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.925582 (+ 46us) service_pool.cc:225] Handling call
0623 08:04:47.604627 (+5679045us) ts_tablet_manager.cc:1145] Deleting tablet a414144276f6484ea4690204752e2142
0623 08:04:47.604638 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.604710 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5678876}
W20250623 08:04:47.605118 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 413) could send its response
I20250623 08:04:47.605252 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 414) took 5679 ms. Trace:
I20250623 08:04:47.605351 952 rpcz_store.cc:276] 0623 08:04:41.925707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.925775 (+ 68us) service_pool.cc:225] Handling call
0623 08:04:47.605140 (+5679365us) ts_tablet_manager.cc:1145] Deleting tablet a25bdc9b7e48467fba3771365e58bcbc
0623 08:04:47.605149 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.605235 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5678400}
W20250623 08:04:47.605589 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 414) could send its response
I20250623 08:04:47.605728 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 416) took 5679 ms. Trace:
I20250623 08:04:47.605830 952 rpcz_store.cc:276] 0623 08:04:41.925962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.926009 (+ 47us) service_pool.cc:225] Handling call
0623 08:04:47.605616 (+5679607us) ts_tablet_manager.cc:1145] Deleting tablet 965f1f0a107b4653b6aac01e0d8e259a
0623 08:04:47.605626 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.605713 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5678803}
W20250623 08:04:47.606052 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 416) could send its response
I20250623 08:04:47.606194 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 419) took 5679 ms. Trace:
I20250623 08:04:47.606289 952 rpcz_store.cc:276] 0623 08:04:41.926235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.926301 (+ 66us) service_pool.cc:225] Handling call
0623 08:04:47.606079 (+5679778us) ts_tablet_manager.cc:1145] Deleting tablet 501b5233522f43c7aed3c9770adb5c03
0623 08:04:47.606087 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.606179 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5679236}
W20250623 08:04:47.606528 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 419) could send its response
I20250623 08:04:47.606678 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 417) took 5680 ms. Trace:
I20250623 08:04:47.606797 952 rpcz_store.cc:276] 0623 08:04:41.926060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.929988 (+ 3928us) service_pool.cc:225] Handling call
0623 08:04:47.606568 (+5676580us) ts_tablet_manager.cc:1145] Deleting tablet 9a0bbafe1ea34ab5ae1b3de25b50d4a3
0623 08:04:47.606576 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.606662 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5676456}
W20250623 08:04:47.607033 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 417) could send its response
I20250623 08:04:47.607180 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 423) took 5636 ms. Trace:
I20250623 08:04:47.607278 952 rpcz_store.cc:276] 0623 08:04:41.970972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.971225 (+ 253us) service_pool.cc:225] Handling call
0623 08:04:47.607065 (+5635840us) ts_tablet_manager.cc:1145] Deleting tablet 8ffafd10f26740c9a5b6c9bdc3bb23a0
0623 08:04:47.607073 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.607163 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5635656}
W20250623 08:04:47.607512 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 423) could send its response
I20250623 08:04:47.607885 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 425) took 5636 ms. Trace:
I20250623 08:04:47.608202 952 rpcz_store.cc:276] 0623 08:04:41.971179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.971445 (+ 266us) service_pool.cc:225] Handling call
0623 08:04:47.607542 (+5636097us) ts_tablet_manager.cc:1145] Deleting tablet d0b2af698b6e4ed0ae66a9bb32be7b31
0623 08:04:47.607552 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.607870 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5635947}
W20250623 08:04:47.608719 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 425) could send its response
W20250623 08:04:47.609239 32589 leader_election.cc:343] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 6c3dcdcaa2e14cb2bd02ebb8f1d0e159
W20250623 08:04:47.610325 32589 consensus_peers.cc:487] T 9049e3d9b39f420bb51221808409fc54 P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.618832 339 ts_tablet_manager.cc:1428] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250623 08:04:47.619163 339 tablet_bootstrap.cc:492] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:47.620786 339 tablet_bootstrap.cc:654] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.625630 1230 catalog_manager.cc:5582] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.630026 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:47.630738 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:47.631112 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:47.631314 1230 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
I20250623 08:04:47.631542 1230 catalog_manager.cc:5582] T 344ebe1a14bc4d3389b0161ff102f05c P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.632066 1230 catalog_manager.cc:5582] T 8e17e1ca404b462da43aa1db7dc3b556 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.632292 2014 consensus_queue.cc:1035] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:47.633030 1225 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2c990297f554adbb2d97f6309758940 consensus state has no leader
I20250623 08:04:47.633203 1225 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57678 (request call id 36) took 1752 ms. Trace:
I20250623 08:04:47.633301 1225 rpcz_store.cc:276] 0623 08:04:45.880722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.199849 (+1319127us) service_pool.cc:225] Handling call
0623 08:04:47.200168 (+ 319us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:47.200365 (+ 197us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:47.200565 (+ 200us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:47.200630 (+ 65us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:47.362136 (+161506us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:47.633164 (+271028us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.205054 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.205140 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.205146 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:47.205148 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:47.205246 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.205255 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.205259 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:47.205307 (+ 48us) tablet.cc:699] Row locks acquired
0623 08:04:47.205310 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:47.205335 (+ 25us) write_op.cc:270] Start()
0623 08:04:47.205358 (+ 23us) write_op.cc:276] Timestamp: P: 1750665887205280 usec, L: 0
0623 08:04:47.205361 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.205481 (+ 120us) log.cc:844] Serialized 405 byte log entry
0623 08:04:47.360248 (+154767us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.361242 (+ 994us) write_op.cc:301] APPLY: starting
0623 08:04:47.361254 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.361281 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.361308 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.361341 (+ 33us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.361363 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.361369 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:47.361418 (+ 49us) log.cc:844] Serialized 23 byte log entry
0623 08:04:47.361457 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.361468 (+ 11us) write_op.cc:454] Released schema lock
0623 08:04:47.361473 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":964,"apply.run_cpu_time_us":285,"apply.run_wall_time_us":284,"num_ops":1,"prepare.queue_time_us":3959,"prepare.run_cpu_time_us":472,"prepare.run_wall_time_us":471,"replication_time_us":154872}]]}
I20250623 08:04:47.634214 1227 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet d5f82dc198af45f598bf05c05a4a8090
I20250623 08:04:47.634430 2329 consensus_queue.cc:1035] T f5f58f55070344ff98a3808f70e0c755 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.634822 1227 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57678 (request call id 24) took 1755 ms. Trace:
W20250623 08:04:47.635133 32589 leader_election.cc:343] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:47.635313 32589 consensus_peers.cc:487] T 3136ec837bea4619ae3995ceec9b24d6 P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3136ec837bea4619ae3995ceec9b24d6. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.634917 1227 rpcz_store.cc:276] 0623 08:04:45.879306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.571271 (+691965us) service_pool.cc:225] Handling call
0623 08:04:46.690527 (+119256us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d9306 00007f79d9223956 00007f79d9223685 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:46.743056 (+ 52529us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:46.743198 (+ 142us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:46.743255 (+ 57us) catalog_manager.cc:2799] Locking tablets
0623 08:04:46.821246 (+ 77991us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:47.341296 (+520050us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:47.341340 (+ 44us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:47.634159 (+292819us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:47.634722 (+ 563us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.187774 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.187867 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.187873 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:47.187875 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:47.187975 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.187984 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.187988 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:47.188078 (+ 90us) tablet.cc:699] Row locks acquired
0623 08:04:47.188081 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:47.188107 (+ 26us) write_op.cc:270] Start()
0623 08:04:47.188131 (+ 24us) write_op.cc:276] Timestamp: P: 1750665887188055 usec, L: 0
0623 08:04:47.188134 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.188268 (+ 134us) log.cc:844] Serialized 725 byte log entry
0623 08:04:47.336111 (+147843us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.336962 (+ 851us) write_op.cc:301] APPLY: starting
0623 08:04:47.336970 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.336999 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.337038 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.337081 (+ 43us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.337103 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:47.337110 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:47.337152 (+ 42us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.337187 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.337198 (+ 11us) write_op.cc:454] Released schema lock
0623 08:04:47.337203 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":250864512,"child_traces":[["op",{"apply.queue_time_us":820,"apply.run_cpu_time_us":286,"apply.run_wall_time_us":286,"mutex_wait_us":44,"num_ops":2,"prepare.queue_time_us":365999,"prepare.run_cpu_time_us":719,"prepare.run_wall_time_us":6786,"raft.queue_time_us":526,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":147936,"thread_start_us":21181,"threads_started":2}]]}
I20250623 08:04:47.635744 1227 master_service.cc:948] ReplaceTablet: received request to replace tablet f20b23fd6e5c44f58b1fb2bf617c5704 from {username='slave'} at 127.0.0.1:57678
W20250623 08:04:47.636133 1248 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca5c694856f44201a8aa4314a4f81681 consensus state has no leader
I20250623 08:04:47.636312 1248 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57678 (request call id 31) took 1756 ms. Trace:
I20250623 08:04:47.636391 1248 rpcz_store.cc:276] 0623 08:04:45.880286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.815155 (+934869us) service_pool.cc:225] Handling call
0623 08:04:47.184653 (+369498us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23da5a6 00007f79d9225616 00007f79d9225345 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:47.196600 (+ 11947us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:47.196850 (+ 250us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:47.197033 (+ 183us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:47.197124 (+ 91us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:47.362055 (+164931us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:47.636255 (+274200us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.204356 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.204457 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.204464 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:47.204466 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:47.204590 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.204603 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.204610 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:47.204681 (+ 71us) tablet.cc:699] Row locks acquired
0623 08:04:47.204683 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:47.204729 (+ 46us) write_op.cc:270] Start()
0623 08:04:47.204763 (+ 34us) write_op.cc:276] Timestamp: P: 1750665887204672 usec, L: 0
0623 08:04:47.204767 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.204947 (+ 180us) log.cc:844] Serialized 405 byte log entry
0623 08:04:47.360201 (+155254us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.360849 (+ 648us) write_op.cc:301] APPLY: starting
0623 08:04:47.360861 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.360893 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.360927 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.360965 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.360994 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.361000 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:47.361054 (+ 54us) log.cc:844] Serialized 23 byte log entry
0623 08:04:47.361096 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.361110 (+ 14us) write_op.cc:454] Released schema lock
0623 08:04:47.361116 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.361181 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.361186 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":849064192,"child_traces":[["op",{"apply.queue_time_us":610,"apply.run_cpu_time_us":322,"apply.run_wall_time_us":323,"num_ops":1,"prepare.queue_time_us":6773,"prepare.run_cpu_time_us":666,"prepare.run_wall_time_us":668,"replication_time_us":155418}]]}
I20250623 08:04:47.637099 1248 master_service.cc:948] ReplaceTablet: received request to replace tablet eef663ffdaf141aea5e6269b6b2e1c89 from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:47.638195 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet d5f82dc198af45f598bf05c05a4a8090 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:47.639349 32377 raft_consensus.cc:3058] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.639968 343 tablet_bootstrap.cc:492] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:47.640120 343 ts_tablet_manager.cc:1397] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.192s user 0.014s sys 0.044s
I20250623 08:04:47.640781 343 raft_consensus.cc:357] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.640957 343 raft_consensus.cc:383] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.641029 343 raft_consensus.cc:738] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.641227 343 consensus_queue.cc:260] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.641795 343 ts_tablet_manager.cc:1428] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:47.642073 343 tablet_bootstrap.cc:492] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:47.643510 343 tablet_bootstrap.cc:654] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:47.644161 1213 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet d5f82dc198af45f598bf05c05a4a8090 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5f82dc198af45f598bf05c05a4a8090 already in progress: creating tablet
I20250623 08:04:47.569864 342 tablet_bootstrap.cc:654] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.647370 339 tablet_bootstrap.cc:492] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:47.647560 339 ts_tablet_manager.cc:1397] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250623 08:04:47.648360 339 raft_consensus.cc:357] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.650527 339 raft_consensus.cc:383] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.650628 339 raft_consensus.cc:738] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.650979 339 consensus_queue.cc:260] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.652575 339 ts_tablet_manager.cc:1428] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:47.654016 339 tablet_bootstrap.cc:492] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:47.658164 339 tablet_bootstrap.cc:654] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:47.665338 1213 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ca5c694856f44201a8aa4314a4f81681 (table default.Table_682a42dbf21d43d49ad9d85b958b121c [id=4f81957f51bd40c192ad7c716cbd893f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ca5c694856f44201a8aa4314a4f81681 consensus state has no leader
I20250623 08:04:47.669128 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 427) took 5697 ms. Trace:
I20250623 08:04:47.674345 952 rpcz_store.cc:276] 0623 08:04:41.971376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.971421 (+ 45us) service_pool.cc:225] Handling call
0623 08:04:47.608748 (+5637327us) ts_tablet_manager.cc:1145] Deleting tablet 36c8714cfbd84dd38cf5c0f305ef6843
0623 08:04:47.608757 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.669102 (+ 60345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5637111}
W20250623 08:04:47.674803 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 427) could send its response
I20250623 08:04:47.675204 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 420) took 5704 ms. Trace:
I20250623 08:04:47.675340 952 rpcz_store.cc:276] 0623 08:04:41.970594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.970662 (+ 68us) service_pool.cc:225] Handling call
0623 08:04:47.675090 (+5704428us) ts_tablet_manager.cc:1145] Deleting tablet f4a0b9402a264800a1ea39707c10da0c
0623 08:04:47.675104 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.675186 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5703406}
W20250623 08:04:47.675632 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 420) could send its response
I20250623 08:04:47.675977 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 426) took 5704 ms. Trace:
I20250623 08:04:47.676172 952 rpcz_store.cc:276] 0623 08:04:41.971277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.971706 (+ 429us) service_pool.cc:225] Handling call
0623 08:04:47.675887 (+5704181us) ts_tablet_manager.cc:1145] Deleting tablet d6ed4cb252b1452087f8ee53ec78d985
0623 08:04:47.675899 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.675961 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5704054}
W20250623 08:04:47.676470 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 426) could send its response
I20250623 08:04:47.676782 952 tablet_replica.cc:331] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:47.677374 952 raft_consensus.cc:2241] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:47.679421 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
W20250623 08:04:47.680884 32589 consensus_peers.cc:487] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.682654 32372 raft_consensus.cc:3058] T 8f83a6d453ec4ffe85282b4385df5f11 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.684705 1229 catalog_manager.cc:5582] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:47.685212 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:47.686021 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:47.686291 1229 catalog_manager.cc:5582] T ea77b97018b74708bd6057bd2d65f90b P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:47.689405 1229 catalog_manager.cc:5582] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } }
I20250623 08:04:47.689793 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:47.690220 1229 catalog_manager.cc:5582] T 4cf1f3943a3d49318c435921ebeaaf33 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } }
I20250623 08:04:47.691028 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:47.690547 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet d5f82dc198af45f598bf05c05a4a8090 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:41574
W20250623 08:04:47.694207 32589 leader_election.cc:343] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.694334 32589 leader_election.cc:304] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:47.694554 2242 raft_consensus.cc:2747] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:47.692281 1213 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet d5f82dc198af45f598bf05c05a4a8090 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5f82dc198af45f598bf05c05a4a8090 already in progress: creating tablet
I20250623 08:04:47.693636 1227 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet f20b23fd6e5c44f58b1fb2bf617c5704
I20250623 08:04:47.697077 1229 catalog_manager.cc:5582] T 7dc5e0fbeba4412aa42d116cd8882238 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } }
I20250623 08:04:47.697651 1227 catalog_manager.cc:6652] ReplaceTablet: tablet f20b23fd6e5c44f58b1fb2bf617c5704 deleted and replaced by tablet a43c956160f2404daab21081daf7b2f4
I20250623 08:04:47.697855 1227 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 37) took 1492 ms. Trace:
I20250623 08:04:47.697777 1229 catalog_manager.cc:5582] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.698220 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:47.698494 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:47.698712 1229 catalog_manager.cc:5582] T 55669db7e55042b2bc208ea3218c2d6c P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:47.699203 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:47.699440 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet f20b23fd6e5c44f58b1fb2bf617c5704 with delete_type TABLET_DATA_DELETED (replaced by tablet a43c956160f2404daab21081daf7b2f4) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:47.700228 339 tablet_bootstrap.cc:492] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:47.700414 339 ts_tablet_manager.cc:1397] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.002s
I20250623 08:04:47.700641 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet f20b23fd6e5c44f58b1fb2bf617c5704 with delete_type TABLET_DATA_DELETED (replaced by tablet a43c956160f2404daab21081daf7b2f4) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:47.701323 339 raft_consensus.cc:357] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.701568 339 raft_consensus.cc:383] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.701676 339 raft_consensus.cc:738] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.702034 339 consensus_queue.cc:260] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.702642 339 ts_tablet_manager.cc:1428] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:47.702988 339 tablet_bootstrap.cc:492] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:47.700043 1213 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet f20b23fd6e5c44f58b1fb2bf617c5704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f20b23fd6e5c44f58b1fb2bf617c5704 already in progress: creating tablet
I20250623 08:04:47.698007 1227 rpcz_store.cc:276] 0623 08:04:46.205322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.635643 (+1430321us) service_pool.cc:225] Handling call
0623 08:04:47.697805 (+ 62162us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.665591 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.665710 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.665717 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:47.665720 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:47.665838 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.665852 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.665857 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:47.665935 (+ 78us) tablet.cc:699] Row locks acquired
0623 08:04:47.665937 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:47.665971 (+ 34us) write_op.cc:270] Start()
0623 08:04:47.666010 (+ 39us) write_op.cc:276] Timestamp: P: 1750665887665922 usec, L: 0
0623 08:04:47.666015 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.666192 (+ 177us) log.cc:844] Serialized 622 byte log entry
0623 08:04:47.670640 (+ 4448us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.670890 (+ 250us) write_op.cc:301] APPLY: starting
0623 08:04:47.670904 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.670953 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.671010 (+ 57us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.671177 (+ 167us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.671234 (+ 57us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.671241 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:47.671326 (+ 85us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.671427 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.671449 (+ 22us) write_op.cc:454] Released schema lock
0623 08:04:47.671459 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.671537 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.671543 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":188,"apply.run_cpu_time_us":640,"apply.run_wall_time_us":640,"num_ops":2,"prepare.queue_time_us":2523,"prepare.run_cpu_time_us":704,"prepare.run_wall_time_us":704,"raft.queue_time_us":2349,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":291,"replication_time_us":4598}]]}
I20250623 08:04:47.705008 1227 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_4a19c65595654fa98efc42626b6a9d85" } new_table_name: "default.Table_bb325e4f89c84cf0ae7aa94b67c08817" modify_external_catalogs: true
I20250623 08:04:47.705257 339 tablet_bootstrap.cc:654] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.704444 1229 catalog_manager.cc:5582] T a0b1287cf3b6459eafd032afc77c5824 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } }
W20250623 08:04:47.704440 1213 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet f20b23fd6e5c44f58b1fb2bf617c5704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f20b23fd6e5c44f58b1fb2bf617c5704 already in progress: creating tablet
I20250623 08:04:47.708289 1229 catalog_manager.cc:5582] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:47.708145 32381 raft_consensus.cc:3058] T de743139c4324faba19ff3e5cfac855d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.713858 1248 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet eef663ffdaf141aea5e6269b6b2e1c89
I20250623 08:04:47.716055 1248 catalog_manager.cc:6652] ReplaceTablet: tablet eef663ffdaf141aea5e6269b6b2e1c89 deleted and replaced by tablet 79e548a8f68840f5a2cfa03ee3232913
I20250623 08:04:47.716167 1248 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 38) took 1211 ms. Trace:
W20250623 08:04:47.716171 32589 consensus_peers.cc:487] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.716629 952 raft_consensus.cc:2270] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.716265 1248 rpcz_store.cc:276] 0623 08:04:46.505137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.637002 (+1131865us) service_pool.cc:225] Handling call
0623 08:04:47.716135 (+ 79133us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.704324 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.704460 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.704467 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:47.704471 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:47.704619 (+ 148us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.704633 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.704638 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:47.704733 (+ 95us) tablet.cc:699] Row locks acquired
0623 08:04:47.704735 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:47.705215 (+ 480us) write_op.cc:270] Start()
0623 08:04:47.705253 (+ 38us) write_op.cc:276] Timestamp: P: 1750665887704717 usec, L: 0
0623 08:04:47.705259 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.705481 (+ 222us) log.cc:844] Serialized 656 byte log entry
0623 08:04:47.712682 (+ 7201us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.712844 (+ 162us) write_op.cc:301] APPLY: starting
0623 08:04:47.712863 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.712919 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.712960 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.713082 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.713148 (+ 66us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.713159 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:47.713244 (+ 85us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.713313 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.713336 (+ 23us) write_op.cc:454] Released schema lock
0623 08:04:47.713346 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.713426 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.713432 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":106,"apply.run_cpu_time_us":570,"apply.run_wall_time_us":570,"num_ops":2,"prepare.queue_time_us":39120,"prepare.run_cpu_time_us":850,"prepare.run_wall_time_us":1936,"raft.queue_time_us":57,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":234,"replication_time_us":7389}]]}
I20250623 08:04:47.717022 1248 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_5c7c44264c394c5e8dc536c5b002b373"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:47.717183 952 ts_tablet_manager.cc:1905] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:47.717592 1248 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c7c44264c394c5e8dc536c5b002b373 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:47.719370 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:47.720187 1229 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:47.718675 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet eef663ffdaf141aea5e6269b6b2e1c89 with delete_type TABLET_DATA_DELETED (replaced by tablet 79e548a8f68840f5a2cfa03ee3232913) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:47.723619 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet eef663ffdaf141aea5e6269b6b2e1c89 with delete_type TABLET_DATA_DELETED (replaced by tablet 79e548a8f68840f5a2cfa03ee3232913) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:47.723838 2540 tablet_replica.cc:331] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:47.724012 2540 raft_consensus.cc:2241] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:47.724146 2540 raft_consensus.cc:2270] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.724615 2540 ts_tablet_manager.cc:1905] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.725211 1229 catalog_manager.cc:5582] T cac83131e6854890b803b3d89ded7cf8 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:47.725754 1229 catalog_manager.cc:5582] T de743139c4324faba19ff3e5cfac855d P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:47.726300 1229 catalog_manager.cc:5582] T 8f83a6d453ec4ffe85282b4385df5f11 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:47.682845 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet d5f82dc198af45f598bf05c05a4a8090 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:47.699024 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet f20b23fd6e5c44f58b1fb2bf617c5704 with delete_type TABLET_DATA_DELETED (replaced by tablet a43c956160f2404daab21081daf7b2f4) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:47.700131 1823 raft_consensus.cc:491] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.732034 1823 raft_consensus.cc:513] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.732558 1823 leader_election.cc:290] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:47.733067 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a24019b797467e857b744d701fb1ed" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.733326 32645 raft_consensus.cc:2466] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:47.733704 32453 leader_election.cc:304] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:47.733970 1823 raft_consensus.cc:2802] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:47.733958 1229 catalog_manager.cc:5582] T d0107b41d5c04a5c818f6792f2c18ce5 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } }
I20250623 08:04:47.734355 1823 raft_consensus.cc:491] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:47.734483 1823 raft_consensus.cc:3058] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.734477 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a24019b797467e857b744d701fb1ed" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:47.735164 32453 leader_election.cc:343] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.736711 2242 consensus_queue.cc:1035] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.737812 1823 raft_consensus.cc:513] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.740609 339 tablet_bootstrap.cc:492] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:47.740783 339 ts_tablet_manager.cc:1397] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.002s
I20250623 08:04:47.742180 2540 ts_tablet_manager.cc:1918] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:47.742331 2540 log.cc:1199] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/eef663ffdaf141aea5e6269b6b2e1c89
I20250623 08:04:47.742772 2540 ts_tablet_manager.cc:1939] T eef663ffdaf141aea5e6269b6b2e1c89 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:47.743939 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet eef663ffdaf141aea5e6269b6b2e1c89 (table default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]) successfully deleted
I20250623 08:04:47.746008 339 raft_consensus.cc:357] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.746264 339 raft_consensus.cc:383] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.746369 339 raft_consensus.cc:738] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.746635 339 consensus_queue.cc:260] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.747504 339 ts_tablet_manager.cc:1428] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250623 08:04:47.748031 339 tablet_bootstrap.cc:492] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:47.750208 339 tablet_bootstrap.cc:654] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.753150 1823 leader_election.cc:290] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:47.754160 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a24019b797467e857b744d701fb1ed" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:47.754392 32647 raft_consensus.cc:3058] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.756503 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet eef663ffdaf141aea5e6269b6b2e1c89 with delete_type TABLET_DATA_DELETED (replaced by tablet 79e548a8f68840f5a2cfa03ee3232913) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:47.757557 32647 raft_consensus.cc:2466] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:47.759418 32453 leader_election.cc:304] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:47.762609 1823 raft_consensus.cc:2802] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:47.762928 1823 raft_consensus.cc:695] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:47.763569 1823 consensus_queue.cc:237] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:47.764750 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a24019b797467e857b744d701fb1ed" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:47.765226 981 tablet_replica.cc:331] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:47.765420 981 raft_consensus.cc:2241] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:47.765374 32453 leader_election.cc:343] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.765551 981 raft_consensus.cc:2270] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.766044 981 ts_tablet_manager.cc:1905] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.767349 2446 raft_consensus.cc:491] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.767500 2446 raft_consensus.cc:513] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.768235 2446 leader_election.cc:290] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.769730 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b720fdea539f423ba2d2aa9f89b09ae9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.771040 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b720fdea539f423ba2d2aa9f89b09ae9" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.771936 32314 leader_election.cc:304] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:47.772586 2446 raft_consensus.cc:2747] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.773345 981 ts_tablet_manager.cc:1918] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:47.773499 981 log.cc:1199] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/eef663ffdaf141aea5e6269b6b2e1c89
I20250623 08:04:47.773917 981 ts_tablet_manager.cc:1939] T eef663ffdaf141aea5e6269b6b2e1c89 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:47.775110 1213 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet eef663ffdaf141aea5e6269b6b2e1c89 (table default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]) successfully deleted
I20250623 08:04:47.776474 342 tablet_bootstrap.cc:492] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:47.776718 342 ts_tablet_manager.cc:1397] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.213s user 0.005s sys 0.000s
I20250623 08:04:47.777565 342 raft_consensus.cc:357] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.777887 342 raft_consensus.cc:383] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.778008 342 raft_consensus.cc:738] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.778326 342 consensus_queue.cc:260] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.779090 342 ts_tablet_manager.cc:1428] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:47.786624 1248 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 40) took 1061 ms. Trace:
I20250623 08:04:47.786814 1248 rpcz_store.cc:276] 0623 08:04:46.724706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.716837 (+992131us) service_pool.cc:225] Handling call
0623 08:04:47.717711 (+ 874us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:47.718202 (+ 491us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:47.786334 (+ 68132us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:47.786450 (+ 116us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:47.786573 (+ 123us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.744249 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.744388 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.744396 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:47.744398 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:47.744513 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.744525 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.744530 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:47.744611 (+ 81us) tablet.cc:699] Row locks acquired
0623 08:04:47.744614 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:47.744648 (+ 34us) write_op.cc:270] Start()
0623 08:04:47.744683 (+ 35us) write_op.cc:276] Timestamp: P: 1750665887744592 usec, L: 0
0623 08:04:47.744687 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.744904 (+ 217us) log.cc:844] Serialized 433 byte log entry
0623 08:04:47.784981 (+ 40077us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.785142 (+ 161us) write_op.cc:301] APPLY: starting
0623 08:04:47.785157 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.785198 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.785232 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.785356 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.785387 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:47.785395 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:47.785462 (+ 67us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.785520 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.785536 (+ 16us) write_op.cc:454] Released schema lock
0623 08:04:47.785542 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.785605 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.785609 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":116,"apply.run_cpu_time_us":454,"apply.run_wall_time_us":453,"num_ops":2,"prepare.queue_time_us":25611,"prepare.run_cpu_time_us":794,"prepare.run_wall_time_us":1625,"raft.queue_time_us":42,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":40260}]]}
I20250623 08:04:47.787525 1248 master_service.cc:948] ReplaceTablet: received request to replace tablet 8911308855cb4414be426dbcec592f4a from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:47.788269 342 tablet_bootstrap.cc:492] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:47.796788 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:47.797168 1235 catalog_manager.cc:5582] T 4e06ef06db2643779f93617e27f100dd P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:47.797933 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:47.798277 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:47.798906 1235 catalog_manager.cc:5582] T 6857437f85ab40f6a24e74ce3b5efbdc P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:47.799614 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:47.799991 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:47.800407 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (table default.Table_4b49e65313cd47e5b682a79f0394d475 [id=ad7210d2665f463c8591e79cf62d15dc]). Expected version 2 got 0
I20250623 08:04:47.800712 1235 catalog_manager.cc:5582] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:47.801055 1248 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 8911308855cb4414be426dbcec592f4a
I20250623 08:04:47.801223 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:47.801456 1248 catalog_manager.cc:6652] ReplaceTablet: tablet 8911308855cb4414be426dbcec592f4a deleted and replaced by tablet e4b90ae23c31460e9ce66f4b14b60317
I20250623 08:04:47.801573 1248 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 41) took 1048 ms. Trace:
I20250623 08:04:47.801645 1248 rpcz_store.cc:276] 0623 08:04:46.753453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.787459 (+1034006us) service_pool.cc:225] Handling call
0623 08:04:47.801523 (+ 14064us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.796301 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.796439 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.796445 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:47.796447 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:47.796553 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.796565 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.796569 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:47.796646 (+ 77us) tablet.cc:699] Row locks acquired
0623 08:04:47.796649 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:47.796682 (+ 33us) write_op.cc:270] Start()
0623 08:04:47.796710 (+ 28us) write_op.cc:276] Timestamp: P: 1750665887796621 usec, L: 0
0623 08:04:47.796714 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.796911 (+ 197us) log.cc:844] Serialized 622 byte log entry
0623 08:04:47.800008 (+ 3097us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.800249 (+ 241us) write_op.cc:301] APPLY: starting
0623 08:04:47.800262 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.800301 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.800334 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.800430 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.800476 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.800482 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:47.800542 (+ 60us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.800592 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.800609 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:47.800615 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":191,"apply.run_cpu_time_us":420,"apply.run_wall_time_us":420,"num_ops":2,"prepare.queue_time_us":7673,"raft.queue_time_us":327,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":220,"replication_time_us":3257}]]}
I20250623 08:04:47.801846 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:47.803017 1235 catalog_manager.cc:5582] T c7f759b707ef46df99873b92c694ad6b P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:47.803613 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:47.803754 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 8911308855cb4414be426dbcec592f4a with delete_type TABLET_DATA_DELETED (replaced by tablet e4b90ae23c31460e9ce66f4b14b60317) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:47.803941 1235 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:47.806177 2242 raft_consensus.cc:491] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.806383 32372 raft_consensus.cc:3058] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.806358 2242 raft_consensus.cc:513] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.806941 2242 leader_election.cc:290] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.804976 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet 8911308855cb4414be426dbcec592f4a with delete_type TABLET_DATA_DELETED (replaced by tablet e4b90ae23c31460e9ce66f4b14b60317) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:47.807166 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 8911308855cb4414be426dbcec592f4a with delete_type TABLET_DATA_DELETED (replaced by tablet e4b90ae23c31460e9ce66f4b14b60317) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:47.807386 2540 tablet_replica.cc:331] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:47.807574 2540 raft_consensus.cc:2241] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:47.807693 2540 raft_consensus.cc:2270] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:47.808166 2540 ts_tablet_manager.cc:1905] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:47.809281 342 tablet_bootstrap.cc:654] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:47.811903 1213 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 8911308855cb4414be426dbcec592f4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8911308855cb4414be426dbcec592f4a already in progress: creating tablet
I20250623 08:04:47.814239 1248 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_ff8fd153c41246b482ea6b6e3f24c02f" } modify_external_catalogs: true
I20250623 08:04:47.814462 1248 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_ff8fd153c41246b482ea6b6e3f24c02f" } modify_external_catalogs: true
I20250623 08:04:47.819494 343 tablet_bootstrap.cc:492] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:47.822887 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eebc32fdfa44abcbd6f04a636d02c7c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:47.822880 2329 raft_consensus.cc:491] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.827322 2329 raft_consensus.cc:513] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.827847 2329 leader_election.cc:290] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.828094 343 ts_tablet_manager.cc:1397] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.186s user 0.012s sys 0.020s
I20250623 08:04:47.829082 343 raft_consensus.cc:357] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.832122 343 raft_consensus.cc:383] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.832432 343 raft_consensus.cc:738] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.833061 343 consensus_queue.cc:260] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.834365 343 ts_tablet_manager.cc:1428] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:47.835505 343 tablet_bootstrap.cc:492] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:47.842257 2540 ts_tablet_manager.cc:1918] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:47.849077 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eebc32fdfa44abcbd6f04a636d02c7c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.860302 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:47.862649 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:47.863099 32453 consensus_peers.cc:487] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eef663ffdaf141aea5e6269b6b2e1c89. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.866878 952 ts_tablet_manager.cc:1918] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:47.873273 952 log.cc:1199] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/adc3cf4355c24b7395fcb22af8520346
I20250623 08:04:47.873888 952 ts_tablet_manager.cc:1939] T adc3cf4355c24b7395fcb22af8520346 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:47.875012 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 428) took 5903 ms. Trace:
I20250623 08:04:47.875223 952 rpcz_store.cc:276] 0623 08:04:41.971483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.971962 (+ 479us) service_pool.cc:225] Handling call
0623 08:04:47.676715 (+5704753us) ts_tablet_manager.cc:1145] Deleting tablet adc3cf4355c24b7395fcb22af8520346
0623 08:04:47.676726 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.739377 (+ 62651us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.866726 (+127349us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:47.874986 (+ 8260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5704622}
W20250623 08:04:47.876108 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 428) could send its response
W20250623 08:04:47.868067 32485 consensus_meta.cc:287] T 5930085e18d84823b27791a65750663f P 59f7c614254f42009306b13337f06576: Time spent flushing consensus metadata: real 0.782s user 0.023s sys 0.001s
I20250623 08:04:47.848308 343 tablet_bootstrap.cc:654] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:47.877605 32453 consensus_peers.cc:487] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3740dd120745493688eeb4c1ac311dcb. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:47.877681 2540 log.cc:1199] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/8911308855cb4414be426dbcec592f4a
I20250623 08:04:47.878011 2540 ts_tablet_manager.cc:1939] T 8911308855cb4414be426dbcec592f4a P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:47.861665 32589 leader_election.cc:343] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.870250 2014 raft_consensus.cc:491] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.878525 2014 raft_consensus.cc:513] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:47.878674 32589 leader_election.cc:343] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.879102 2014 leader_election.cc:290] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:47.880018 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 8911308855cb4414be426dbcec592f4a (table default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22]) successfully deleted
I20250623 08:04:47.868798 342 tablet_bootstrap.cc:492] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:47.880743 342 ts_tablet_manager.cc:1397] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.093s user 0.004s sys 0.000s
I20250623 08:04:47.881704 342 raft_consensus.cc:357] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:47.888326 32589 leader_election.cc:343] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.888785 342 raft_consensus.cc:383] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.888926 32589 leader_election.cc:304] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:47.889122 342 raft_consensus.cc:738] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.889763 342 consensus_queue.cc:260] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.890414 2558 raft_consensus.cc:2747] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.895182 343 tablet_bootstrap.cc:492] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:47.895357 343 ts_tablet_manager.cc:1397] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.007s
I20250623 08:04:47.896031 343 raft_consensus.cc:357] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.896236 343 raft_consensus.cc:383] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.896364 343 raft_consensus.cc:738] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.896612 343 consensus_queue.cc:260] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.897137 343 ts_tablet_manager.cc:1428] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:47.897429 343 tablet_bootstrap.cc:492] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:47.898989 343 tablet_bootstrap.cc:654] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:47.899581 32589 leader_election.cc:343] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.899721 32589 leader_election.cc:304] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:47.901448 342 ts_tablet_manager.cc:1428] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250623 08:04:47.901937 2558 raft_consensus.cc:2747] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.904359 342 tablet_bootstrap.cc:492] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:47.907610 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e4cde2af694f07a9176313b4b93a46" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:47.907930 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e4cde2af694f07a9176313b4b93a46" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:47.876015 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 424) took 5904 ms. Trace:
I20250623 08:04:47.908345 952 rpcz_store.cc:276] 0623 08:04:41.971059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.972684 (+ 1625us) service_pool.cc:225] Handling call
0623 08:04:47.875923 (+5903239us) ts_tablet_manager.cc:1145] Deleting tablet 6c6f98e2db1a4b0789c5819b2c7d4bb6
0623 08:04:47.875932 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.875999 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5903032}
W20250623 08:04:47.908567 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 424) could send its response
I20250623 08:04:47.910352 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 430) took 5938 ms. Trace:
I20250623 08:04:47.910422 952 rpcz_store.cc:276] 0623 08:04:41.972152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:41.972968 (+ 816us) service_pool.cc:225] Handling call
0623 08:04:47.910248 (+5937280us) ts_tablet_manager.cc:1145] Deleting tablet ba5913811e9343389c9ea0ec8b8c05ef
0623 08:04:47.910257 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:47.910338 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5937124}
W20250623 08:04:47.910607 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 430) could send its response
I20250623 08:04:47.911728 952 tablet_replica.cc:331] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:47.916492 32372 raft_consensus.cc:3058] T 4cf1f3943a3d49318c435921ebeaaf33 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.912669 952 raft_consensus.cc:2241] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: Raft consensus shutting down.
I20250623 08:04:47.924713 952 pending_rounds.cc:70] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: Trying to abort 1 pending ops.
I20250623 08:04:47.926391 952 pending_rounds.cc:77] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7170727472914812928 op_type: NO_OP noop_request { }
I20250623 08:04:47.927915 952 raft_consensus.cc:2887] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250623 08:04:47.928804 952 raft_consensus.cc:2270] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:47.914801 32589 leader_election.cc:343] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.925990 342 tablet_bootstrap.cc:654] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:47.935772 32589 leader_election.cc:343] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.940186 32589 leader_election.cc:304] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:47.943459 2558 raft_consensus.cc:2747] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:47.946226 343 tablet_bootstrap.cc:492] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:47.946429 343 ts_tablet_manager.cc:1397] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.012s
I20250623 08:04:47.947228 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 3a66ee8ae6aa43d689189e16b223e8e8 (DEFAULT_TABLE table=default.Table_6867d69edd3b45a3b76992f22cf0207a [id=a5639a1f1b0746b09e5d3b70dece2e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:47.947624 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a66ee8ae6aa43d689189e16b223e8e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:47.955018 2612 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.69.62:39465 every 8 connection(s)
I20250623 08:04:47.958930 2197 rpc_server.cc:307] RPC server started. Bound to: 127.31.69.62:39465
I20250623 08:04:47.960868 343 raft_consensus.cc:357] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.961037 343 raft_consensus.cc:383] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:47.961086 343 raft_consensus.cc:738] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:47.961328 2197 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/master-0/data/info.pb
I20250623 08:04:47.961272 343 consensus_queue.cc:260] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.961719 343 ts_tablet_manager.cc:1428] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20250623 08:04:47.962054 343 tablet_bootstrap.cc:492] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:47.963831 343 tablet_bootstrap.cc:654] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:47.964579 32020 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskJxinPB/build/asan/bin/kudu as pid 2197
I20250623 08:04:47.981524 1957 raft_consensus.cc:491] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:47.981721 1957 raft_consensus.cc:513] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.982340 1957 leader_election.cc:290] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:47.983120 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecc73f52f0bc43c5bbdca2b89deecb21" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:47.983359 32647 raft_consensus.cc:2466] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:47.983875 32453 leader_election.cc:304] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:47.984196 1957 raft_consensus.cc:2802] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:47.984283 1957 raft_consensus.cc:491] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:47.984367 1957 raft_consensus.cc:3058] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.987818 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecc73f52f0bc43c5bbdca2b89deecb21" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:47.987833 1957 raft_consensus.cc:513] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:47.988375 1957 leader_election.cc:290] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:47.989899 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecc73f52f0bc43c5bbdca2b89deecb21" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
W20250623 08:04:47.988878 32453 leader_election.cc:343] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:47.990077 32646 raft_consensus.cc:3058] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:47.990499 1957 consensus_queue.cc:1035] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:47.993016 32646 raft_consensus.cc:2466] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:47.994773 32453 leader_election.cc:304] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:47.995002 1823 raft_consensus.cc:2802] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:47.995121 1823 raft_consensus.cc:695] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:47.995401 1823 consensus_queue.cc:237] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.001186 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecc73f52f0bc43c5bbdca2b89deecb21" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:48.001780 32453 leader_election.cc:343] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:48.003862 32589 consensus_peers.cc:487] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bfa787f828b411ab68d81bb896f7fb6. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:48.009321 2620 tablet.cc:1722] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:48.043419 339 tablet_bootstrap.cc:492] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.043643 339 ts_tablet_manager.cc:1397] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.296s user 0.016s sys 0.043s
I20250623 08:04:48.044430 339 raft_consensus.cc:357] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.044682 339 raft_consensus.cc:383] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.044775 339 raft_consensus.cc:738] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.045012 339 consensus_queue.cc:260] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.054095 952 ts_tablet_manager.cc:1905] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.057582 2014 raft_consensus.cc:491] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.057762 2014 raft_consensus.cc:513] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.058468 2014 leader_election.cc:290] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.059067 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36d21fe528f5410eacd655bf15306b98" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:48.059580 32589 leader_election.cc:343] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.060956 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36d21fe528f5410eacd655bf15306b98" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:48.062367 32589 leader_election.cc:343] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.062526 32589 leader_election.cc:304] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.062920 2014 raft_consensus.cc:2747] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.063143 2583 tablet.cc:1722] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:48.079041 2578 tablet.cc:1722] T 5af1dea3c6ca4f498ce6c039d72a14b7 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:48.080072 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 3a66ee8ae6aa43d689189e16b223e8e8 (DEFAULT_TABLE table=default.Table_6867d69edd3b45a3b76992f22cf0207a [id=a5639a1f1b0746b09e5d3b70dece2e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.081072 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 3a66ee8ae6aa43d689189e16b223e8e8 (DEFAULT_TABLE table=default.Table_6867d69edd3b45a3b76992f22cf0207a [id=a5639a1f1b0746b09e5d3b70dece2e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.081408 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a66ee8ae6aa43d689189e16b223e8e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.085914 339 ts_tablet_manager.cc:1428] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20250623 08:04:48.086205 339 tablet_bootstrap.cc:492] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.096859 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 8f04cba8c9da4d9581eb9a0f05bd027e (DEFAULT_TABLE table=default.Table_2efaa95bf059477fb226d82901892c43 [id=ea222ca127a84e63b7fbf4764e5a98c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.097255 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f04cba8c9da4d9581eb9a0f05bd027e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.102037 32427 heartbeater.cc:344] Connected to a master server at 127.31.69.62:39465
I20250623 08:04:48.102221 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.103816 32567 heartbeater.cc:344] Connected to a master server at 127.31.69.62:39465
I20250623 08:04:48.104022 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.110970 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 8f04cba8c9da4d9581eb9a0f05bd027e (DEFAULT_TABLE table=default.Table_2efaa95bf059477fb226d82901892c43 [id=ea222ca127a84e63b7fbf4764e5a98c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.111260 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 8f04cba8c9da4d9581eb9a0f05bd027e (DEFAULT_TABLE table=default.Table_2efaa95bf059477fb226d82901892c43 [id=ea222ca127a84e63b7fbf4764e5a98c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.113117 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 8e7fe7f600584b259423875312b85bfd (DEFAULT_TABLE table=default.Table_d6a3e0399f3b420c821b3393e20499df [id=db582cdd12d245c5b66fe9e6ba6c25d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.114005 32629 tablet_service.cc:1468] Processing CreateTablet for tablet 8e7fe7f600584b259423875312b85bfd (DEFAULT_TABLE table=default.Table_d6a3e0399f3b420c821b3393e20499df [id=db582cdd12d245c5b66fe9e6ba6c25d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.114145 32346 tablet_service.cc:1468] Processing CreateTablet for tablet 502d98366eeb4d5eb87c2a2dcbd86f59 (DEFAULT_TABLE table=default.Table_bbc25f3d36674be8bd76f8f90a2f74b8 [id=14a0c431f8204262bb57d6a3d9987ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.114530 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 502d98366eeb4d5eb87c2a2dcbd86f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.115284 343 tablet_bootstrap.cc:492] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:48.115512 343 ts_tablet_manager.cc:1397] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.154s user 0.002s sys 0.011s
I20250623 08:04:48.116165 343 raft_consensus.cc:357] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.116315 343 raft_consensus.cc:383] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.116359 343 raft_consensus.cc:738] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.116554 343 consensus_queue.cc:260] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.117292 343 ts_tablet_manager.cc:1428] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:48.121304 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 8e7fe7f600584b259423875312b85bfd (DEFAULT_TABLE table=default.Table_d6a3e0399f3b420c821b3393e20499df [id=db582cdd12d245c5b66fe9e6ba6c25d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.121694 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7fe7f600584b259423875312b85bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.126853 342 tablet_bootstrap.cc:492] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.126991 342 ts_tablet_manager.cc:1397] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.224s user 0.001s sys 0.008s
I20250623 08:04:48.127880 342 raft_consensus.cc:357] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.128129 342 raft_consensus.cc:383] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.128226 342 raft_consensus.cc:738] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.128533 342 consensus_queue.cc:260] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.129153 342 ts_tablet_manager.cc:1428] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:48.129535 342 tablet_bootstrap.cc:492] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.135416 339 tablet_bootstrap.cc:654] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.138775 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7fe7f600584b259423875312b85bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.139374 343 tablet_bootstrap.cc:492] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:48.140892 343 tablet_bootstrap.cc:654] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.144810 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f04cba8c9da4d9581eb9a0f05bd027e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.158037 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f04cba8c9da4d9581eb9a0f05bd027e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.158219 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e7fe7f600584b259423875312b85bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.162474 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a66ee8ae6aa43d689189e16b223e8e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.165009 1823 raft_consensus.cc:491] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.165130 1823 raft_consensus.cc:513] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.165745 1823 leader_election.cc:290] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.166592 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "185c2bf7b9254947bb76b7fe0924559e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:48.166849 32645 raft_consensus.cc:2466] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:48.167501 32453 leader_election.cc:304] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:48.168452 1823 raft_consensus.cc:2802] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:48.168536 1823 raft_consensus.cc:491] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:48.168609 1823 raft_consensus.cc:3058] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.171175 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "185c2bf7b9254947bb76b7fe0924559e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.171418 32368 raft_consensus.cc:2466] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:48.172276 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 502d98366eeb4d5eb87c2a2dcbd86f59 (DEFAULT_TABLE table=default.Table_bbc25f3d36674be8bd76f8f90a2f74b8 [id=14a0c431f8204262bb57d6a3d9987ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.172752 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 502d98366eeb4d5eb87c2a2dcbd86f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.175818 342 tablet_bootstrap.cc:654] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.177289 1823 raft_consensus.cc:513] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.177842 1823 leader_election.cc:290] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.178606 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "185c2bf7b9254947bb76b7fe0924559e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:48.178853 32645 raft_consensus.cc:3058] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.183259 32645 raft_consensus.cc:2466] T 185c2bf7b9254947bb76b7fe0924559e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:48.184007 32453 leader_election.cc:304] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:48.184689 1953 raft_consensus.cc:2802] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:48.184839 1953 raft_consensus.cc:695] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:48.185081 32346 tablet_service.cc:1468] Processing CreateTablet for tablet ebbac9a9cc8348ec9b1b25f7fb9c9d1a (DEFAULT_TABLE table=default.Table_71fd58f8442b4035bf100aa5edc058da [id=32ad6dcb6bf14bb694675269d2a6732c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.185068 1953 consensus_queue.cc:237] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.185508 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebbac9a9cc8348ec9b1b25f7fb9c9d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.190925 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "185c2bf7b9254947bb76b7fe0924559e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:48.191129 32377 raft_consensus.cc:3058] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.193619 32377 raft_consensus.cc:2466] T 185c2bf7b9254947bb76b7fe0924559e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:48.194125 32500 tablet_service.cc:1468] Processing CreateTablet for tablet ebbac9a9cc8348ec9b1b25f7fb9c9d1a (DEFAULT_TABLE table=default.Table_71fd58f8442b4035bf100aa5edc058da [id=32ad6dcb6bf14bb694675269d2a6732c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.194698 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebbac9a9cc8348ec9b1b25f7fb9c9d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.207239 32629 tablet_service.cc:1468] Processing CreateTablet for tablet ebbac9a9cc8348ec9b1b25f7fb9c9d1a (DEFAULT_TABLE table=default.Table_71fd58f8442b4035bf100aa5edc058da [id=32ad6dcb6bf14bb694675269d2a6732c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.207739 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebbac9a9cc8348ec9b1b25f7fb9c9d1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.215139 32357 tablet_service.cc:1468] Processing CreateTablet for tablet 19136569bef9483481ad37f794b1e411 (DEFAULT_TABLE table=default.Table_69a26f395448447eb6d121e1f5520160 [id=af3854442e874946b610cb17b5aa2069]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.215651 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 19136569bef9483481ad37f794b1e411 (DEFAULT_TABLE table=default.Table_69a26f395448447eb6d121e1f5520160 [id=af3854442e874946b610cb17b5aa2069]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.215690 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 19136569bef9483481ad37f794b1e411 (DEFAULT_TABLE table=default.Table_69a26f395448447eb6d121e1f5520160 [id=af3854442e874946b610cb17b5aa2069]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.216008 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19136569bef9483481ad37f794b1e411. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.216008 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19136569bef9483481ad37f794b1e411. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.225841 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.226768 32345 tablet_service.cc:1468] Processing CreateTablet for tablet e270c441f47c4a5b80213766c096e369 (DEFAULT_TABLE table=default.Table_58ecf0b93bc44862912280870cdcd018 [id=adead3f2715949e6b876059be77199bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.227108 32499 tablet_service.cc:1468] Processing CreateTablet for tablet e270c441f47c4a5b80213766c096e369 (DEFAULT_TABLE table=default.Table_58ecf0b93bc44862912280870cdcd018 [id=adead3f2715949e6b876059be77199bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.227505 32499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e270c441f47c4a5b80213766c096e369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.227486 32633 tablet_service.cc:1468] Processing CreateTablet for tablet e270c441f47c4a5b80213766c096e369 (DEFAULT_TABLE table=default.Table_58ecf0b93bc44862912280870cdcd018 [id=adead3f2715949e6b876059be77199bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.227828 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e270c441f47c4a5b80213766c096e369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.231307 32348 tablet_service.cc:1468] Processing CreateTablet for tablet 06fe16d2aa1c4a059e91348c40410bc2 (DEFAULT_TABLE table=default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe [id=f77e622d3eb44f4fa3a284b08eefc109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.232455 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 06fe16d2aa1c4a059e91348c40410bc2 (DEFAULT_TABLE table=default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe [id=f77e622d3eb44f4fa3a284b08eefc109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.232900 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06fe16d2aa1c4a059e91348c40410bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.233536 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 859d5270cd2f4eaf89dc95631931d8de (DEFAULT_TABLE table=default.Table_20d45edbd83549339391048931491a56 [id=56fa63b2cd564946846905d99b29b258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.233861 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 859d5270cd2f4eaf89dc95631931d8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.237301 2337 raft_consensus.cc:491] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.237406 2337 raft_consensus.cc:513] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.237919 2337 leader_election.cc:290] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.238454 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15154c85de7144a9b9e2367c70f57e2e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:48.238946 32589 leader_election.cc:343] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.239010 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 859d5270cd2f4eaf89dc95631931d8de (DEFAULT_TABLE table=default.Table_20d45edbd83549339391048931491a56 [id=56fa63b2cd564946846905d99b29b258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.154712 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.242022 32486 tablet_service.cc:1468] Processing CreateTablet for tablet 06fe16d2aa1c4a059e91348c40410bc2 (DEFAULT_TABLE table=default.Table_ff0d3d8fbb884f4e911e8b43d0d0b9fe [id=f77e622d3eb44f4fa3a284b08eefc109]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.242384 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06fe16d2aa1c4a059e91348c40410bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.245271 343 tablet_bootstrap.cc:492] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:48.245934 339 tablet_bootstrap.cc:492] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.246115 343 ts_tablet_manager.cc:1397] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.005s
I20250623 08:04:48.246289 339 ts_tablet_manager.cc:1397] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.160s user 0.004s sys 0.004s
I20250623 08:04:48.248042 32702 heartbeater.cc:344] Connected to a master server at 127.31.69.62:39465
I20250623 08:04:48.249452 952 ts_tablet_manager.cc:1918] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250623 08:04:48.249549 952 log.cc:1199] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/3740dd120745493688eeb4c1ac311dcb
I20250623 08:04:48.249891 952 ts_tablet_manager.cc:1939] T 3740dd120745493688eeb4c1ac311dcb P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:48.250196 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 442) took 5801 ms. Trace:
I20250623 08:04:48.250264 952 rpcz_store.cc:276] 0623 08:04:42.448558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.448602 (+ 44us) service_pool.cc:225] Handling call
0623 08:04:47.911688 (+5463086us) ts_tablet_manager.cc:1145] Deleting tablet 3740dd120745493688eeb4c1ac311dcb
0623 08:04:47.911697 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.164122 (+252425us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.249346 (+ 85224us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.250181 (+ 835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5421571}
W20250623 08:04:48.250524 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 442) could send its response
I20250623 08:04:48.250599 952 tablet_replica.cc:331] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:48.254932 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 26a368f05593414e9e5b1c8d1c70c106 (DEFAULT_TABLE table=default.Table_0a60238bfb2e432d94faeeb3792b594a [id=53577e6a44684f7cbc9d064bd9ea0a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.255383 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26a368f05593414e9e5b1c8d1c70c106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.257292 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 859d5270cd2f4eaf89dc95631931d8de (DEFAULT_TABLE table=default.Table_20d45edbd83549339391048931491a56 [id=56fa63b2cd564946846905d99b29b258]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.254671 32342 tablet_service.cc:1468] Processing CreateTablet for tablet 26a368f05593414e9e5b1c8d1c70c106 (DEFAULT_TABLE table=default.Table_0a60238bfb2e432d94faeeb3792b594a [id=53577e6a44684f7cbc9d064bd9ea0a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.259446 343 raft_consensus.cc:357] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.266422 339 raft_consensus.cc:357] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.259892 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 859d5270cd2f4eaf89dc95631931d8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.270803 343 raft_consensus.cc:383] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.270948 343 raft_consensus.cc:738] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.271194 343 consensus_queue.cc:260] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.271632 32633 tablet_service.cc:1468] Processing CreateTablet for tablet 26a368f05593414e9e5b1c8d1c70c106 (DEFAULT_TABLE table=default.Table_0a60238bfb2e432d94faeeb3792b594a [id=53577e6a44684f7cbc9d064bd9ea0a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.271871 343 ts_tablet_manager.cc:1428] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250623 08:04:48.272049 339 raft_consensus.cc:383] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.272650 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26a368f05593414e9e5b1c8d1c70c106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.273789 32626 tablet_service.cc:1468] Processing CreateTablet for tablet 502d98366eeb4d5eb87c2a2dcbd86f59 (DEFAULT_TABLE table=default.Table_bbc25f3d36674be8bd76f8f90a2f74b8 [id=14a0c431f8204262bb57d6a3d9987ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.273872 339 raft_consensus.cc:738] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.274211 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 502d98366eeb4d5eb87c2a2dcbd86f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.274187 339 consensus_queue.cc:260] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.277360 343 tablet_bootstrap.cc:492] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:48.277648 339 ts_tablet_manager.cc:1428] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250623 08:04:48.279198 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:48.279860 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:48.280005 343 tablet_bootstrap.cc:654] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.280197 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:48.280752 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:48.281082 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:48.281252 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:48.281373 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:48.281533 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (table default.Table_4b49e65313cd47e5b682a79f0394d475 [id=ad7210d2665f463c8591e79cf62d15dc]). Expected version 2 got 0
I20250623 08:04:48.283375 342 tablet_bootstrap.cc:492] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.283650 342 ts_tablet_manager.cc:1397] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.154s user 0.003s sys 0.004s
I20250623 08:04:48.284840 342 raft_consensus.cc:357] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.285153 342 raft_consensus.cc:383] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.285295 342 raft_consensus.cc:738] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.285705 342 consensus_queue.cc:260] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.286697 342 ts_tablet_manager.cc:1428] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:48.287535 342 tablet_bootstrap.cc:492] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.293110 342 tablet_bootstrap.cc:654] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.297441 343 tablet_bootstrap.cc:492] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:48.297649 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15154c85de7144a9b9e2367c70f57e2e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.297760 343 ts_tablet_manager.cc:1397] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20250623 08:04:48.298373 32374 raft_consensus.cc:3058] T 55669db7e55042b2bc208ea3218c2d6c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.298867 32380 raft_consensus.cc:3058] T c7f759b707ef46df99873b92c694ad6b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.299116 343 raft_consensus.cc:357] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.299360 343 raft_consensus.cc:383] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.299429 343 raft_consensus.cc:738] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.299657 32428 heartbeater.cc:499] Master 127.31.69.61:33313 was elected leader, sending a full tablet report...
I20250623 08:04:48.299827 343 consensus_queue.cc:260] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:48.300038 32589 leader_election.cc:343] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.300166 32589 leader_election.cc:304] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.300457 2517 raft_consensus.cc:2747] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.301350 343 ts_tablet_manager.cc:1428] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:48.302968 343 tablet_bootstrap.cc:492] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:48.305626 32589 consensus_peers.cc:487] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c3dcdcaa2e14cb2bd02ebb8f1d0e159. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:48.307121 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19136569bef9483481ad37f794b1e411. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.307228 339 tablet_bootstrap.cc:492] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.308143 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e270c441f47c4a5b80213766c096e369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.308176 1359 raft_consensus.cc:491] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.308311 1359 raft_consensus.cc:513] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.308563 339 tablet_bootstrap.cc:654] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.308812 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06fe16d2aa1c4a059e91348c40410bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.308815 1359 leader_election.cc:290] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.309801 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 859d5270cd2f4eaf89dc95631931d8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.310344 32342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26a368f05593414e9e5b1c8d1c70c106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.312512 343 tablet_bootstrap.cc:654] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.312944 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3576624314d4e85b411aab9109683f7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.313323 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3576624314d4e85b411aab9109683f7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:48.313488 32647 raft_consensus.cc:2466] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:48.313675 32453 consensus_peers.cc:487] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9ea74d20c5f4eefbcb8508ed1774026. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:48.314826 32453 leader_election.cc:343] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.317024 32453 leader_election.cc:304] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.317242 1359 raft_consensus.cc:2802] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:48.317315 1359 raft_consensus.cc:491] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:48.317375 1359 raft_consensus.cc:3058] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.320429 1359 raft_consensus.cc:513] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.321003 1359 leader_election.cc:290] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.322589 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3576624314d4e85b411aab9109683f7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:48.323025 952 raft_consensus.cc:2241] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:48.323024 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3576624314d4e85b411aab9109683f7" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:48.323196 32647 raft_consensus.cc:3058] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.323396 952 raft_consensus.cc:2270] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.323961 952 ts_tablet_manager.cc:1905] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.325825 32647 raft_consensus.cc:2466] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:48.329631 32453 leader_election.cc:343] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.329936 32453 leader_election.cc:304] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.329907 343 tablet_bootstrap.cc:492] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:48.330170 1823 raft_consensus.cc:2802] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:48.330276 1823 raft_consensus.cc:695] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:48.330420 343 ts_tablet_manager.cc:1397] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.001s
I20250623 08:04:48.330564 1823 consensus_queue.cc:237] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.333807 343 raft_consensus.cc:357] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.334645 343 raft_consensus.cc:383] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.335032 343 raft_consensus.cc:738] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.338879 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:48.339309 1359 raft_consensus.cc:491] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.339466 1359 raft_consensus.cc:513] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.340202 1359 leader_election.cc:290] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.340221 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.341205 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
I20250623 08:04:48.342130 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7b9e843ff1344a1aaacf038f15f64cd" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:48.342314 32647 raft_consensus.cc:2466] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:48.343436 32453 leader_election.cc:304] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:48.343739 1953 raft_consensus.cc:2802] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:48.343819 1953 raft_consensus.cc:491] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:48.343883 1953 raft_consensus.cc:3058] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.346673 1953 raft_consensus.cc:513] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.347436 1953 leader_election.cc:290] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.348802 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7b9e843ff1344a1aaacf038f15f64cd" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:48.349180 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7b9e843ff1344a1aaacf038f15f64cd" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:48.349346 32647 raft_consensus.cc:3058] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.349422 2517 raft_consensus.cc:491] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.349553 2517 raft_consensus.cc:513] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.350086 2517 leader_election.cc:290] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.350665 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b8cfcbce6a472ab2daae28226cf766" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:48.351110 32589 leader_election.cc:343] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.351959 32647 raft_consensus.cc:2466] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:48.352638 339 tablet_bootstrap.cc:492] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.352639 32453 leader_election.cc:304] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:48.352873 1359 raft_consensus.cc:2802] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:48.353014 1359 raft_consensus.cc:695] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:48.352944 339 ts_tablet_manager.cc:1397] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20250623 08:04:48.353333 1359 consensus_queue.cc:237] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.354259 339 raft_consensus.cc:357] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.354566 339 raft_consensus.cc:383] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.354707 339 raft_consensus.cc:738] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.355132 339 consensus_queue.cc:260] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.356138 339 ts_tablet_manager.cc:1428] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:48.356599 339 tablet_bootstrap.cc:492] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.357141 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b8cfcbce6a472ab2daae28226cf766" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:48.358314 32453 consensus_peers.cc:487] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:48.359364 339 tablet_bootstrap.cc:654] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:48.359586 32453 leader_election.cc:343] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:48.360109 32589 leader_election.cc:343] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.360236 32589 leader_election.cc:304] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.360700 2517 raft_consensus.cc:2747] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:48.372200 32453 consensus_peers.cc:487] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb06a340f82047ac8ec3516edb08f4b6. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:48.374431 32453 consensus_peers.cc:487] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eef663ffdaf141aea5e6269b6b2e1c89. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:48.347697 343 consensus_queue.cc:260] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.382117 343 ts_tablet_manager.cc:1428] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.051s user 0.003s sys 0.000s
I20250623 08:04:48.385561 342 tablet_bootstrap.cc:492] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.385886 342 ts_tablet_manager.cc:1397] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.099s user 0.011s sys 0.008s
I20250623 08:04:48.387702 342 raft_consensus.cc:357] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.388067 342 raft_consensus.cc:383] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.388196 342 raft_consensus.cc:738] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.388571 342 consensus_queue.cc:260] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.389415 343 tablet_bootstrap.cc:492] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:48.390568 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7b9e843ff1344a1aaacf038f15f64cd" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.390986 339 tablet_bootstrap.cc:492] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
W20250623 08:04:48.391075 32453 leader_election.cc:343] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.391834 342 ts_tablet_manager.cc:1428] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:48.391782 339 ts_tablet_manager.cc:1397] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.000s
I20250623 08:04:48.393318 342 tablet_bootstrap.cc:492] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.395061 339 raft_consensus.cc:357] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.395773 339 raft_consensus.cc:383] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.396100 339 raft_consensus.cc:738] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.396901 339 consensus_queue.cc:260] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.398489 339 ts_tablet_manager.cc:1428] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:48.399426 339 tablet_bootstrap.cc:492] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.399392 342 tablet_bootstrap.cc:654] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.407995 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:48.409539 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:48.412940 343 tablet_bootstrap.cc:654] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.413530 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:48.413714 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:48.413919 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:48.414090 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:48.428547 2613 sys_catalog.cc:263] Verifying existing consensus state
I20250623 08:04:48.432346 2446 raft_consensus.cc:491] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.432539 2446 raft_consensus.cc:513] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.433337 2446 leader_election.cc:290] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.433696 2517 raft_consensus.cc:491] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.433830 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f5f58f55070344ff98a3808f70e0c755" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:48.433822 2517 raft_consensus.cc:513] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.434340 2517 leader_election.cc:290] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.434582 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f5f58f55070344ff98a3808f70e0c755" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:48.435555 32314 leader_election.cc:304] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:48.435782 2446 raft_consensus.cc:2747] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.436175 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:48.436333 32510 raft_consensus.cc:2466] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:48.436683 32589 leader_election.cc:304] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:48.437005 2517 raft_consensus.cc:2802] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:48.437084 2517 raft_consensus.cc:491] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:48.437157 2517 raft_consensus.cc:3058] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:48.437383 952 ts_tablet_manager.cc:1918] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:48.437469 952 log.cc:1199] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/1be8f8557b9a4ae2917c37b8d01cedf6
I20250623 08:04:48.437800 952 ts_tablet_manager.cc:1939] T 1be8f8557b9a4ae2917c37b8d01cedf6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:48.438076 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 441) took 5989 ms. Trace:
I20250623 08:04:48.438139 952 rpcz_store.cc:276] 0623 08:04:42.448404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.448456 (+ 52us) service_pool.cc:225] Handling call
0623 08:04:48.250559 (+5802103us) ts_tablet_manager.cc:1145] Deleting tablet 1be8f8557b9a4ae2917c37b8d01cedf6
0623 08:04:48.250568 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.432110 (+181542us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.437296 (+ 5186us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.438062 (+ 766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5760317}
I20250623 08:04:48.438424 952 tablet_replica.cc:331] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:48.438587 952 raft_consensus.cc:2241] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:48.439780 2517 raft_consensus.cc:513] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
W20250623 08:04:48.440335 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 441) could send its response
I20250623 08:04:48.441207 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:48.441344 32510 raft_consensus.cc:3058] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:48.442831 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:48.442981 32374 raft_consensus.cc:3058] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:48.443885 32510 raft_consensus.cc:2466] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 2.
I20250623 08:04:48.444897 32589 leader_election.cc:304] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:48.445114 2337 raft_consensus.cc:2802] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:48.445199 2337 raft_consensus.cc:695] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [term 2 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:48.445412 2337 consensus_queue.cc:237] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.446031 32374 raft_consensus.cc:2466] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 2.
I20250623 08:04:48.446477 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:48.447477 2517 leader_election.cc:290] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.449630 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b97f3019b893441bae18146fef094bd3" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.450109 32379 raft_consensus.cc:2374] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 2. Re-sending same reply.
I20250623 08:04:48.453894 2646 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9752155c20ee42fbb7ff7b7d5b682444)
I20250623 08:04:48.454156 2646 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:48.456728 2517 raft_consensus.cc:491] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.456887 2517 raft_consensus.cc:513] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.458191 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 51fce46407ac4e57a74ad696e85daac7 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:48.458561 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30dfa986dfd44658a0ffb07796030520" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:48.458897 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
W20250623 08:04:48.459087 32589 leader_election.cc:343] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.459261 2517 leader_election.cc:290] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.459470 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:48.459512 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30dfa986dfd44658a0ffb07796030520" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.459668 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
W20250623 08:04:48.459998 32589 leader_election.cc:343] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.460101 32589 leader_election.cc:304] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.460369 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c50655c06cd94c67a49513c39728eaa9 (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:48.460507 2517 raft_consensus.cc:2747] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.460939 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:48.461349 343 tablet_bootstrap.cc:492] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:48.461540 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:48.461755 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:48.462495 343 ts_tablet_manager.cc:1397] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.004s
I20250623 08:04:48.465763 343 raft_consensus.cc:357] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.466411 343 raft_consensus.cc:383] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.466667 2646 leader_election.cc:290] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313)
I20250623 08:04:48.466790 343 raft_consensus.cc:738] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.467589 1267 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 5 candidate_status { last_received { term: 4 index: 796 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
I20250623 08:04:48.469066 342 tablet_bootstrap.cc:492] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.469370 342 ts_tablet_manager.cc:1397] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.006s
I20250623 08:04:48.470625 342 raft_consensus.cc:357] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.470968 342 raft_consensus.cc:383] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.471105 342 raft_consensus.cc:738] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.471524 342 consensus_queue.cc:260] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.467528 343 consensus_queue.cc:260] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.473176 342 ts_tablet_manager.cc:1428] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:48.475364 342 tablet_bootstrap.cc:492] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.476040 343 ts_tablet_manager.cc:1428] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250623 08:04:48.477779 343 tablet_bootstrap.cc:492] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:48.486450 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1cee3de5953244f98428747bea912f77 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]). Expected version 1 got 0
I20250623 08:04:48.486552 342 tablet_bootstrap.cc:654] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.488211 343 tablet_bootstrap.cc:654] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.489017 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:48.489279 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:48.491242 32379 raft_consensus.cc:1273] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Refusing update from remote peer 125745849172449ba4be8244d61fdcb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 08:04:48.491797 2337 consensus_queue.cc:1035] T 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:48.493456 339 tablet_bootstrap.cc:654] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.500505 2613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Bootstrap starting.
I20250623 08:04:48.504909 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.507550 952 raft_consensus.cc:2270] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.508153 952 ts_tablet_manager.cc:1905] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.516813 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_8f362905029746c19e0e7ce1433c594d [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 2 got 0
I20250623 08:04:48.517346 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:48.517539 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7f890ee0747b4df0b82bc3b396191124 (table default.Table_7ccb5edd6b824f308d349c51d926d963 [id=9f97f6deca194137a353ea221d0e3676]). Expected version 1 got 0
I20250623 08:04:48.517761 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 035bb06bc7dd42f681886def9d56eb73 (table default.Table_53879d6f821f4167bf02ffbe967f73f7 [id=b3f407a7a26e4bd5ae9a719a5c4577c0]). Expected version 1 got 0
I20250623 08:04:48.517897 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ca5c694856f44201a8aa4314a4f81681 (table default.Table_682a42dbf21d43d49ad9d85b958b121c [id=4f81957f51bd40c192ad7c716cbd893f]). Expected version 1 got 0
I20250623 08:04:48.518081 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1c529e826fa84a919c018887620e4023 (table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]). Expected version 1 got 0
I20250623 08:04:48.518249 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:48.518504 339 tablet_bootstrap.cc:492] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.518667 339 ts_tablet_manager.cc:1397] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.007s
I20250623 08:04:48.518714 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:48.519428 339 raft_consensus.cc:357] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.519636 339 raft_consensus.cc:383] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.519724 339 raft_consensus.cc:738] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.519986 339 consensus_queue.cc:260] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.520538 339 ts_tablet_manager.cc:1428] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:48.520905 339 tablet_bootstrap.cc:492] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.522403 339 tablet_bootstrap.cc:654] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.522562 343 tablet_bootstrap.cc:492] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:48.523167 343 ts_tablet_manager.cc:1397] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.004s
I20250623 08:04:48.526630 1823 raft_consensus.cc:491] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.526782 1823 raft_consensus.cc:513] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.527405 1823 leader_election.cc:290] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.527539 32374 raft_consensus.cc:3058] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.527594 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "297bdf6c0e424190a35b55cbeefe7161" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.529956 343 raft_consensus.cc:357] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.531067 343 raft_consensus.cc:383] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 08:04:48.531365 32453 leader_election.cc:343] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.531694 1823 consensus_queue.cc:1035] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:48.531987 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "297bdf6c0e424190a35b55cbeefe7161" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:48.532398 32647 raft_consensus.cc:2466] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:48.532159 343 raft_consensus.cc:738] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.533304 32453 leader_election.cc:304] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.533608 1823 raft_consensus.cc:2802] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:48.533680 1823 raft_consensus.cc:491] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:48.533741 1823 raft_consensus.cc:3058] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.536339 1823 raft_consensus.cc:513] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.537094 1823 leader_election.cc:290] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.537655 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "297bdf6c0e424190a35b55cbeefe7161" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:48.537819 32647 raft_consensus.cc:3058] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.538111 32368 raft_consensus.cc:3058] T b720fdea539f423ba2d2aa9f89b09ae9 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.539041 342 tablet_bootstrap.cc:492] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.539266 342 ts_tablet_manager.cc:1397] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.005s
I20250623 08:04:48.532228 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:48.540401 32647 raft_consensus.cc:2466] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:48.540771 32453 leader_election.cc:304] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:48.541009 1823 raft_consensus.cc:2802] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:48.541112 1823 raft_consensus.cc:695] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:48.541309 1823 consensus_queue.cc:237] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.542150 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "297bdf6c0e424190a35b55cbeefe7161" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:48.543006 32453 leader_election.cc:343] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.540544 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:48.540740 342 raft_consensus.cc:357] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.546234 342 raft_consensus.cc:383] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.546571 342 raft_consensus.cc:738] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.547334 342 consensus_queue.cc:260] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.548810 343 consensus_queue.cc:260] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.551761 342 ts_tablet_manager.cc:1428] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20250623 08:04:48.546598 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:48.554318 343 ts_tablet_manager.cc:1428] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20250623 08:04:48.554523 342 tablet_bootstrap.cc:492] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.555822 339 tablet_bootstrap.cc:492] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.557663 339 ts_tablet_manager.cc:1397] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.001s
I20250623 08:04:48.558166 1225 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:48.560667 952 ts_tablet_manager.cc:1918] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:48.562163 1248 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 1f76ef24c69f4b17a373b242d71da8b9
I20250623 08:04:48.562495 339 raft_consensus.cc:357] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.562664 339 raft_consensus.cc:383] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.562716 339 raft_consensus.cc:738] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.562738 1248 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57678 (request call id 42) took 1784 ms. Trace:
I20250623 08:04:48.563038 339 consensus_queue.cc:260] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.562872 1248 rpcz_store.cc:276] 0623 08:04:46.778308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.802281 (+1023973us) service_pool.cc:225] Handling call
0623 08:04:47.814109 (+ 11828us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d9306 00007f79d9223956 00007f79d9223685 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:47.814589 (+ 480us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:47.814834 (+ 245us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:47.814919 (+ 85us) catalog_manager.cc:2799] Locking tablets
0623 08:04:47.815105 (+ 186us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:47.851606 (+ 36501us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:47.851705 (+ 99us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:48.562094 (+710389us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:48.562687 (+ 593us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.836909 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.837003 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.837008 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:47.837010 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:47.837110 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.837120 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.837125 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:47.842519 (+ 5394us) tablet.cc:699] Row locks acquired
0623 08:04:47.842522 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:47.842555 (+ 33us) write_op.cc:270] Start()
0623 08:04:47.842584 (+ 29us) write_op.cc:276] Timestamp: P: 1750665887842489 usec, L: 0
0623 08:04:47.842588 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.842774 (+ 186us) log.cc:844] Serialized 759 byte log entry
0623 08:04:47.850576 (+ 7802us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.850734 (+ 158us) write_op.cc:301] APPLY: starting
0623 08:04:47.850772 (+ 38us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.850829 (+ 57us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.850862 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.850946 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.850991 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:47.851003 (+ 12us) write_op.cc:312] APPLY: finished
0623 08:04:47.851088 (+ 85us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.851155 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.851181 (+ 26us) write_op.cc:454] Released schema lock
0623 08:04:47.851190 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.851286 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.851292 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":26497920,"child_traces":[["op",{"apply.queue_time_us":92,"apply.run_cpu_time_us":535,"apply.run_wall_time_us":535,"num_ops":2,"prepare.queue_time_us":20940,"prepare.run_cpu_time_us":822,"prepare.run_wall_time_us":12804,"raft.queue_time_us":546,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":331,"replication_time_us":7953}]]}
I20250623 08:04:48.563632 339 ts_tablet_manager.cc:1428] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250623 08:04:48.563910 339 tablet_bootstrap.cc:492] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.564536 2446 raft_consensus.cc:491] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.564646 2446 raft_consensus.cc:513] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.565217 2446 leader_election.cc:290] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.567471 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f76ef24c69f4b17a373b242d71da8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:47 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:48.571523 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74c1cdc83ffd4a09a163479ac58fd405" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:48.571758 32647 raft_consensus.cc:2391] T 74c1cdc83ffd4a09a163479ac58fd405 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:48.572939 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f76ef24c69f4b17a373b242d71da8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:47 UTC) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:48.574463 339 tablet_bootstrap.cc:654] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.575855 343 tablet_bootstrap.cc:492] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:48.578598 2659 tablet_replica.cc:331] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:48.585891 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.587877 2659 raft_consensus.cc:2241] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:48.579077 343 tablet_bootstrap.cc:654] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.588385 1227 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57678 (request call id 39) took 1865 ms. Trace:
I20250623 08:04:48.588716 2659 raft_consensus.cc:2270] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.588495 1227 rpcz_store.cc:276] 0623 08:04:46.722449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.704812 (+982363us) service_pool.cc:225] Handling call
0623 08:04:47.705243 (+ 431us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:47.705854 (+ 611us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:47.706021 (+ 167us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:47.706110 (+ 89us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:47.779717 (+ 73607us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:48.588327 (+808610us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.709361 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.709484 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.709490 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:47.709492 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:47.709572 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.709581 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.709585 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:47.709634 (+ 49us) tablet.cc:699] Row locks acquired
0623 08:04:47.709637 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:47.709664 (+ 27us) write_op.cc:270] Start()
0623 08:04:47.709690 (+ 26us) write_op.cc:276] Timestamp: P: 1750665887709614 usec, L: 0
0623 08:04:47.709694 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.709852 (+ 158us) log.cc:844] Serialized 405 byte log entry
0623 08:04:47.778910 (+ 69058us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.779090 (+ 180us) write_op.cc:301] APPLY: starting
0623 08:04:47.779111 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.779160 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.779209 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.779267 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.779312 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.779324 (+ 12us) write_op.cc:312] APPLY: finished
0623 08:04:47.779416 (+ 92us) log.cc:844] Serialized 23 byte log entry
0623 08:04:47.779485 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.779505 (+ 20us) write_op.cc:454] Released schema lock
0623 08:04:47.779514 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.779600 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.779606 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":121,"apply.run_cpu_time_us":497,"apply.run_wall_time_us":496,"num_ops":1,"prepare.queue_time_us":2768,"prepare.run_cpu_time_us":592,"prepare.run_wall_time_us":1147,"raft.queue_time_us":33,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":69172}]]}
I20250623 08:04:48.589891 2659 ts_tablet_manager.cc:1905] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.597092 892 master_runner.cc:186] Error getting master registration for 127.31.69.62:39465: OK, instance_id { permanent_uuid: "d31835ec4ad74332a00998d431492466" instance_seqno: 1750665887928376 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250623 08:04:48.597383 892 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250623 08:04:48.601236 2337 raft_consensus.cc:491] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.601368 2337 raft_consensus.cc:513] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.601881 343 tablet_bootstrap.cc:492] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:48.601990 2337 leader_election.cc:290] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.602125 343 ts_tablet_manager.cc:1397] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.001s
I20250623 08:04:48.603031 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4e01b430a064740b76e0bceec4f5546" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.603487 339 tablet_bootstrap.cc:492] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.603677 339 ts_tablet_manager.cc:1397] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
W20250623 08:04:48.603647 32589 leader_election.cc:343] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.603904 343 raft_consensus.cc:357] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.604449 343 raft_consensus.cc:383] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.604486 339 raft_consensus.cc:357] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.604599 343 raft_consensus.cc:738] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.604760 339 raft_consensus.cc:383] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.604905 339 raft_consensus.cc:738] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.605006 343 consensus_queue.cc:260] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.605263 339 consensus_queue.cc:260] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.605947 339 ts_tablet_manager.cc:1428] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:48.606253 339 tablet_bootstrap.cc:492] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.607555 343 ts_tablet_manager.cc:1428] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250623 08:04:48.608847 339 tablet_bootstrap.cc:654] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.565621 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74c1cdc83ffd4a09a163479ac58fd405" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:48.609342 343 tablet_bootstrap.cc:492] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:48.571309 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f76ef24c69f4b17a373b242d71da8b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:47 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:48.575232 342 tablet_bootstrap.cc:654] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.596607 2439 consensus_queue.cc:1035] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:48.577527 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.598942 1359 raft_consensus.cc:491] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.609905 1359 raft_consensus.cc:513] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.610404 1359 leader_election.cc:290] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.604142 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4e01b430a064740b76e0bceec4f5546" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:48.611207 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b213c8c685c43868df1e814c79fc61f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:48.611287 32589 leader_election.cc:343] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.611368 32646 raft_consensus.cc:2466] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:48.611392 32589 leader_election.cc:304] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.611589 2337 raft_consensus.cc:2747] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.612000 32453 leader_election.cc:304] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters:
I20250623 08:04:48.612246 1359 raft_consensus.cc:2802] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:48.612320 1359 raft_consensus.cc:491] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:48.612372 1359 raft_consensus.cc:3058] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.610291 32314 leader_election.cc:304] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:48.612943 2446 raft_consensus.cc:3058] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.613025 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b213c8c685c43868df1e814c79fc61f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.615475 2446 raft_consensus.cc:2747] T 74c1cdc83ffd4a09a163479ac58fd405 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.615263 1359 raft_consensus.cc:513] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:48.615937 32453 leader_election.cc:343] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.616626 1359 leader_election.cc:290] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.616760 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b213c8c685c43868df1e814c79fc61f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:48.618026 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b213c8c685c43868df1e814c79fc61f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:48.618185 32646 raft_consensus.cc:3058] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.620811 32646 raft_consensus.cc:2466] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:48.621415 32453 leader_election.cc:343] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.622011 32453 leader_election.cc:304] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.622790 1359 raft_consensus.cc:2802] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:48.623152 1359 raft_consensus.cc:695] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:48.623605 1359 consensus_queue.cc:237] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.624822 32702 heartbeater.cc:344] Connected to a master server at 127.31.69.62:39465
I20250623 08:04:48.625941 952 log.cc:1199] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a9ea74d20c5f4eefbcb8508ed1774026
I20250623 08:04:48.625963 2659 ts_tablet_manager.cc:1918] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:48.631013 2659 log.cc:1199] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/1f76ef24c69f4b17a373b242d71da8b9
I20250623 08:04:48.631542 2659 ts_tablet_manager.cc:1939] T 1f76ef24c69f4b17a373b242d71da8b9 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:48.632342 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 1f76ef24c69f4b17a373b242d71da8b9 (table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]) successfully deleted
I20250623 08:04:48.637352 952 ts_tablet_manager.cc:1939] T a9ea74d20c5f4eefbcb8508ed1774026 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:48.646605 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 444) took 6181 ms. Trace:
I20250623 08:04:48.646895 952 rpcz_store.cc:276] 0623 08:04:42.465577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.467585 (+ 2008us) service_pool.cc:225] Handling call
0623 08:04:48.438384 (+5970799us) ts_tablet_manager.cc:1145] Deleting tablet a9ea74d20c5f4eefbcb8508ed1774026
0623 08:04:48.438392 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.511339 (+ 72947us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.560552 (+ 49213us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.646581 (+ 86029us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5947518}
I20250623 08:04:48.651674 2660 tablet_replica.cc:331] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:48.652001 2660 raft_consensus.cc:2241] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:48.655149 952 tablet_replica.cc:331] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:48.665064 2660 raft_consensus.cc:2270] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.666121 2660 ts_tablet_manager.cc:1905] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.666781 952 raft_consensus.cc:2241] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus shutting down.
W20250623 08:04:48.655326 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 444) could send its response
I20250623 08:04:48.668466 952 raft_consensus.cc:2270] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.669173 952 ts_tablet_manager.cc:1905] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.674177 2337 raft_consensus.cc:491] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.674319 2337 raft_consensus.cc:513] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.674930 2337 leader_election.cc:290] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.676111 2439 consensus_queue.cc:1035] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:48.677702 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcfe68d4e6ae470d816ac4c494dcaa51" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:48.683369 32589 leader_election.cc:343] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:48.692297 32453 consensus_peers.cc:487] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:48.695953 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcfe68d4e6ae470d816ac4c494dcaa51" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:48.696549 32589 leader_election.cc:343] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.696724 32589 leader_election.cc:304] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.697149 2337 raft_consensus.cc:2747] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.703104 2578 tablet.cc:1722] T 71fadb10e6134478b013358bba056389 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:48.707664 2660 ts_tablet_manager.cc:1918] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:48.709069 2660 log.cc:1199] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/1f76ef24c69f4b17a373b242d71da8b9
I20250623 08:04:48.709115 343 tablet_bootstrap.cc:654] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.714921 342 tablet_bootstrap.cc:492] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.715281 342 ts_tablet_manager.cc:1397] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.162s user 0.008s sys 0.004s
I20250623 08:04:48.724946 2660 ts_tablet_manager.cc:1939] T 1f76ef24c69f4b17a373b242d71da8b9 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:48.728928 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.729856 343 tablet_bootstrap.cc:492] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:48.730053 343 ts_tablet_manager.cc:1397] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.121s user 0.004s sys 0.001s
I20250623 08:04:48.732139 343 raft_consensus.cc:357] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.732391 343 raft_consensus.cc:383] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.732491 343 raft_consensus.cc:738] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.732760 343 consensus_queue.cc:260] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.733316 343 ts_tablet_manager.cc:1428] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:48.733640 343 tablet_bootstrap.cc:492] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:48.735728 1213 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 1f76ef24c69f4b17a373b242d71da8b9 (table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]) successfully deleted
I20250623 08:04:48.739540 339 tablet_bootstrap.cc:492] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.739748 339 ts_tablet_manager.cc:1397] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.134s user 0.010s sys 0.006s
I20250623 08:04:48.740489 339 raft_consensus.cc:357] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.740715 339 raft_consensus.cc:383] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.740834 339 raft_consensus.cc:738] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.741088 339 consensus_queue.cc:260] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.744568 339 ts_tablet_manager.cc:1428] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:48.744903 339 tablet_bootstrap.cc:492] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.746289 339 tablet_bootstrap.cc:654] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.734687 1823 raft_consensus.cc:491] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.748173 952 ts_tablet_manager.cc:1918] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:48.749009 952 log.cc:1199] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a8de2b1105564fdab386442b1de4cedb
I20250623 08:04:48.748168 342 raft_consensus.cc:357] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.749228 342 raft_consensus.cc:383] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.749292 342 raft_consensus.cc:738] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.749356 952 ts_tablet_manager.cc:1939] T a8de2b1105564fdab386442b1de4cedb P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:48.749464 342 consensus_queue.cc:260] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.749893 342 ts_tablet_manager.cc:1428] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.034s user 0.001s sys 0.001s
I20250623 08:04:48.748989 1823 raft_consensus.cc:513] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.760803 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.760984 1823 leader_election.cc:290] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.766757 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 443) took 6301 ms. Trace:
I20250623 08:04:48.766928 342 tablet_bootstrap.cc:492] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.766865 952 rpcz_store.cc:276] 0623 08:04:42.465446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:42.474294 (+ 8848us) service_pool.cc:225] Handling call
0623 08:04:48.655084 (+6180790us) ts_tablet_manager.cc:1145] Deleting tablet a8de2b1105564fdab386442b1de4cedb
0623 08:04:48.655098 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.673910 (+ 18812us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.748068 (+ 74158us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.766715 (+ 18647us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85,"tablet-delete.queue_time_us":6164116}
I20250623 08:04:48.767326 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 0) took 2149 ms. Trace:
I20250623 08:04:48.767397 952 rpcz_store.cc:276] 0623 08:04:46.617638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.618133 (+ 495us) service_pool.cc:225] Handling call
0623 08:04:48.767216 (+2149083us) ts_tablet_manager.cc:1145] Deleting tablet dcfe68d4e6ae470d816ac4c494dcaa51
0623 08:04:48.767224 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.767311 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2124548}
I20250623 08:04:48.767665 952 tablet_replica.cc:331] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:48.767879 952 raft_consensus.cc:2241] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:48.768028 952 raft_consensus.cc:2270] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.768512 952 ts_tablet_manager.cc:1905] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.768607 342 tablet_bootstrap.cc:654] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:48.771159 1213 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet dcfe68d4e6ae470d816ac4c494dcaa51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcfe68d4e6ae470d816ac4c494dcaa51 already in progress: creating tablet
I20250623 08:04:48.777725 952 ts_tablet_manager.cc:1918] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:48.779093 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf753ede17f429b82612ad12e13185c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:48.779995 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf753ede17f429b82612ad12e13185c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:48.780210 32654 raft_consensus.cc:2466] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:48.782270 952 log.cc:1199] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/ea588a48e6164d3db148fdaa306268f6
I20250623 08:04:48.782727 952 ts_tablet_manager.cc:1939] T ea588a48e6164d3db148fdaa306268f6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:48.783121 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 1) took 2164 ms. Trace:
I20250623 08:04:48.783252 952 rpcz_store.cc:276] 0623 08:04:46.618520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.618951 (+ 431us) service_pool.cc:225] Handling call
0623 08:04:48.767614 (+2148663us) ts_tablet_manager.cc:1145] Deleting tablet ea588a48e6164d3db148fdaa306268f6
0623 08:04:48.767624 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.773626 (+ 6002us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.777635 (+ 4009us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.783104 (+ 5469us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2124855}
I20250623 08:04:48.783819 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet ea588a48e6164d3db148fdaa306268f6 (table default.Table_b523bec7182443139e891357d295e3a1 [id=1c4298a5f9014d8eb4ccad8c8fbdeeac]) successfully deleted
I20250623 08:04:48.784198 952 tablet_replica.cc:331] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:48.784968 952 raft_consensus.cc:2241] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:48.786696 2517 raft_consensus.cc:491] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.784682 2673 tablet.cc:1722] T 71fadb10e6134478b013358bba056389 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:48.786845 2517 raft_consensus.cc:513] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:48.787119 1218 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 035bb06bc7dd42f681886def9d56eb73 (table default.Table_53879d6f821f4167bf02ffbe967f73f7 [id=b3f407a7a26e4bd5ae9a719a5c4577c0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 035bb06bc7dd42f681886def9d56eb73 consensus state has no leader
I20250623 08:04:48.787410 2517 leader_election.cc:290] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.787859 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee82db25aa544a21a1419e081196f179" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:48.788915 32589 leader_election.cc:343] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.792436 342 tablet_bootstrap.cc:492] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.792513 952 raft_consensus.cc:2270] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.792743 342 ts_tablet_manager.cc:1397] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.002s
I20250623 08:04:48.793218 952 ts_tablet_manager.cc:1905] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:48.779626 32453 connection.cc:441] server connection from 127.0.0.1:55526 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55526 (request call id 443) could send its response
I20250623 08:04:48.804316 343 tablet_bootstrap.cc:654] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:48.804745 32453 leader_election.cc:343] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.805006 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee82db25aa544a21a1419e081196f179" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:48.805172 32453 leader_election.cc:304] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.805423 2439 raft_consensus.cc:2802] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:48.805495 2439 raft_consensus.cc:491] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:48.805555 2439 raft_consensus.cc:3058] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.805979 2446 raft_consensus.cc:491] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.806103 2446 raft_consensus.cc:513] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.806659 2446 leader_election.cc:290] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.810178 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e06ef06db2643779f93617e27f100dd" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:48.811504 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e06ef06db2643779f93617e27f100dd" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:48.812081 32314 leader_election.cc:304] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:48.812346 339 tablet_bootstrap.cc:492] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.812420 2446 raft_consensus.cc:2747] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.812462 339 ts_tablet_manager.cc:1397] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.000s
I20250623 08:04:48.794169 342 raft_consensus.cc:357] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.812681 2439 raft_consensus.cc:513] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.812883 342 raft_consensus.cc:383] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.813007 342 raft_consensus.cc:738] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.813526 2439 leader_election.cc:290] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.817421 339 raft_consensus.cc:357] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.813364 342 consensus_queue.cc:260] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.821362 342 ts_tablet_manager.cc:1428] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20250623 08:04:48.821730 342 tablet_bootstrap.cc:492] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.821856 339 raft_consensus.cc:383] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.822089 339 raft_consensus.cc:738] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.823210 339 consensus_queue.cc:260] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:48.824641 32589 leader_election.cc:343] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:48.824769 32589 leader_election.cc:304] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:48.825224 2558 raft_consensus.cc:2747] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.825771 342 tablet_bootstrap.cc:654] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.827464 2679 tablet.cc:1722] T 71fadb10e6134478b013358bba056389 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:48.825299 339 ts_tablet_manager.cc:1428] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250623 08:04:48.830807 339 tablet_bootstrap.cc:492] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.832294 339 tablet_bootstrap.cc:654] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.832630 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf753ede17f429b82612ad12e13185c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:48.832806 32647 raft_consensus.cc:3058] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.833559 2558 raft_consensus.cc:491] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:48.833693 2558 raft_consensus.cc:513] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.834194 2558 leader_election.cc:290] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.835613 32647 raft_consensus.cc:2466] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:48.832361 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf753ede17f429b82612ad12e13185c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:48.837432 32589 leader_election.cc:343] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 4b11777e0bd04e61a5bc8bb6ec4d97c3
I20250623 08:04:48.837549 952 ts_tablet_manager.cc:1918] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:48.837668 952 log.cc:1199] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/bb06a340f82047ac8ec3516edb08f4b6
I20250623 08:04:48.838042 952 ts_tablet_manager.cc:1939] T bb06a340f82047ac8ec3516edb08f4b6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:48.847231 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.854954 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 2) took 2235 ms. Trace:
I20250623 08:04:48.856298 952 rpcz_store.cc:276] 0623 08:04:46.619420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.619490 (+ 70us) service_pool.cc:225] Handling call
0623 08:04:48.784153 (+2164663us) ts_tablet_manager.cc:1145] Deleting tablet bb06a340f82047ac8ec3516edb08f4b6
0623 08:04:48.784163 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.814551 (+ 30388us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.837437 (+ 22886us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.854927 (+ 17490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2141317}
I20250623 08:04:48.856817 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 22) took 1504 ms. Trace:
I20250623 08:04:48.856868 32342 tablet_service.cc:1468] Processing CreateTablet for tablet f6cb10c7ecd54f00b725cf484c12c379 (DEFAULT_TABLE table=default.Table_5c7c44264c394c5e8dc536c5b002b373 [id=93bae7c8e1394ddbbdd1256d9013e436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.856972 952 rpcz_store.cc:276] 0623 08:04:47.352146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.353686 (+ 1540us) service_pool.cc:225] Handling call
0623 08:04:48.856707 (+1503021us) ts_tablet_manager.cc:1145] Deleting tablet adc3cf4355c24b7395fcb22af8520346
0623 08:04:48.856717 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.856798 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502764}
I20250623 08:04:48.857313 32342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6cb10c7ecd54f00b725cf484c12c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.857455 952 tablet_replica.cc:331] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:48.857434 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 79e548a8f68840f5a2cfa03ee3232913 (DEFAULT_TABLE table=default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.857532 32350 tablet_service.cc:1468] Processing CreateTablet for tablet a43c956160f2404daab21081daf7b2f4 (DEFAULT_TABLE table=default.Table_744bf9c83e004aafb60d2cb44530dff5 [id=dd185047529940ebbd12e27a9f6577a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.857695 952 raft_consensus.cc:2241] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:48.857800 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e548a8f68840f5a2cfa03ee3232913. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.857854 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43c956160f2404daab21081daf7b2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.858006 952 raft_consensus.cc:2270] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.858520 952 ts_tablet_manager.cc:1905] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.862824 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 79e548a8f68840f5a2cfa03ee3232913 (DEFAULT_TABLE table=default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.863015 32348 tablet_service.cc:1468] Processing CreateTablet for tablet e4b90ae23c31460e9ce66f4b14b60317 (DEFAULT_TABLE table=default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.863188 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e548a8f68840f5a2cfa03ee3232913. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.863338 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b90ae23c31460e9ce66f4b14b60317. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.863675 32626 tablet_service.cc:1468] Processing CreateTablet for tablet a43c956160f2404daab21081daf7b2f4 (DEFAULT_TABLE table=default.Table_744bf9c83e004aafb60d2cb44530dff5 [id=dd185047529940ebbd12e27a9f6577a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.863993 32626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43c956160f2404daab21081daf7b2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.866994 32633 tablet_service.cc:1468] Processing CreateTablet for tablet e4b90ae23c31460e9ce66f4b14b60317 (DEFAULT_TABLE table=default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.867321 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b90ae23c31460e9ce66f4b14b60317. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.868938 32629 tablet_service.cc:1468] Processing CreateTablet for tablet f6cb10c7ecd54f00b725cf484c12c379 (DEFAULT_TABLE table=default.Table_5c7c44264c394c5e8dc536c5b002b373 [id=93bae7c8e1394ddbbdd1256d9013e436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.869271 32629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6cb10c7ecd54f00b725cf484c12c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.871769 32486 tablet_service.cc:1468] Processing CreateTablet for tablet e4b90ae23c31460e9ce66f4b14b60317 (DEFAULT_TABLE table=default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.871924 32485 tablet_service.cc:1468] Processing CreateTablet for tablet f6cb10c7ecd54f00b725cf484c12c379 (DEFAULT_TABLE table=default.Table_5c7c44264c394c5e8dc536c5b002b373 [id=93bae7c8e1394ddbbdd1256d9013e436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.872130 32486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4b90ae23c31460e9ce66f4b14b60317. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.872373 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 79e548a8f68840f5a2cfa03ee3232913 (DEFAULT_TABLE table=default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.872673 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e548a8f68840f5a2cfa03ee3232913. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.873006 339 tablet_bootstrap.cc:492] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.873119 339 ts_tablet_manager.cc:1397] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20250623 08:04:48.873713 339 raft_consensus.cc:357] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.873871 339 raft_consensus.cc:383] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.873914 339 raft_consensus.cc:738] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.875133 339 consensus_queue.cc:260] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.875368 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6cb10c7ecd54f00b725cf484c12c379. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.875618 339 ts_tablet_manager.cc:1428] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:48.875885 339 tablet_bootstrap.cc:492] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.876053 342 tablet_bootstrap.cc:492] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.876161 342 ts_tablet_manager.cc:1397] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.008s
I20250623 08:04:48.877189 342 raft_consensus.cc:357] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.877318 339 tablet_bootstrap.cc:654] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.877331 342 raft_consensus.cc:383] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.877383 342 raft_consensus.cc:738] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.877573 342 consensus_queue.cc:260] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.871769 32500 tablet_service.cc:1468] Processing CreateTablet for tablet a43c956160f2404daab21081daf7b2f4 (DEFAULT_TABLE table=default.Table_744bf9c83e004aafb60d2cb44530dff5 [id=dd185047529940ebbd12e27a9f6577a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:48.878129 342 ts_tablet_manager.cc:1428] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:48.878288 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43c956160f2404daab21081daf7b2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:48.878361 342 tablet_bootstrap.cc:492] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.879874 342 tablet_bootstrap.cc:654] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.883342 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet bb06a340f82047ac8ec3516edb08f4b6 (table default.Table_e0696987af214f9fb46f9c76b529fca5 [id=045c96a8828945689bffafd8ccc236ec]) successfully deleted
W20250623 08:04:48.883639 1213 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet adc3cf4355c24b7395fcb22af8520346 because the tablet was not found. No further retry: Not found: Tablet not found: adc3cf4355c24b7395fcb22af8520346
I20250623 08:04:48.887557 342 tablet_bootstrap.cc:492] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.887691 342 ts_tablet_manager.cc:1397] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20250623 08:04:48.888377 342 raft_consensus.cc:357] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.888535 342 raft_consensus.cc:383] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.888597 342 raft_consensus.cc:738] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.889578 342 consensus_queue.cc:260] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.890043 342 ts_tablet_manager.cc:1428] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:48.890306 342 tablet_bootstrap.cc:492] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.891870 342 tablet_bootstrap.cc:654] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.895233 342 tablet_bootstrap.cc:492] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.895354 342 ts_tablet_manager.cc:1397] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250623 08:04:48.895992 342 raft_consensus.cc:357] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.896136 342 raft_consensus.cc:383] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.896194 342 raft_consensus.cc:738] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.896435 342 consensus_queue.cc:260] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.897029 342 ts_tablet_manager.cc:1428] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:48.897287 342 tablet_bootstrap.cc:492] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.897492 339 tablet_bootstrap.cc:492] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:48.897655 339 ts_tablet_manager.cc:1397] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.001s
I20250623 08:04:48.898970 342 tablet_bootstrap.cc:654] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.899473 952 ts_tablet_manager.cc:1918] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:48.899755 339 raft_consensus.cc:357] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.899989 339 raft_consensus.cc:383] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.900122 339 raft_consensus.cc:738] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.900462 339 consensus_queue.cc:260] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.901290 339 ts_tablet_manager.cc:1428] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250623 08:04:48.901819 339 tablet_bootstrap.cc:492] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:48.902823 952 log.cc:1199] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/eef663ffdaf141aea5e6269b6b2e1c89
I20250623 08:04:48.903236 952 ts_tablet_manager.cc:1939] T eef663ffdaf141aea5e6269b6b2e1c89 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:48.903937 339 tablet_bootstrap.cc:654] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.906548 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:48.906929 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:48.903959 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 25) took 1185 ms. Trace:
I20250623 08:04:48.907673 342 tablet_bootstrap.cc:492] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.910979 342 ts_tablet_manager.cc:1397] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20250623 08:04:48.910876 952 rpcz_store.cc:276] 0623 08:04:47.718548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.718606 (+ 58us) service_pool.cc:225] Handling call
0623 08:04:48.857356 (+1138750us) ts_tablet_manager.cc:1145] Deleting tablet eef663ffdaf141aea5e6269b6b2e1c89
0623 08:04:48.857418 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.861476 (+ 4058us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.899379 (+ 37903us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.903942 (+ 4563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134481}
I20250623 08:04:48.911756 342 raft_consensus.cc:357] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.911935 342 raft_consensus.cc:383] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.911995 342 raft_consensus.cc:738] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.912124 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 24) took 1213 ms. Trace:
I20250623 08:04:48.912163 342 consensus_queue.cc:260] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.912235 952 rpcz_store.cc:276] 0623 08:04:47.698926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.698976 (+ 50us) service_pool.cc:225] Handling call
0623 08:04:48.911833 (+1212857us) ts_tablet_manager.cc:1145] Deleting tablet f20b23fd6e5c44f58b1fb2bf617c5704
0623 08:04:48.911842 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.912108 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1179936}
I20250623 08:04:48.912636 342 ts_tablet_manager.cc:1428] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:48.912948 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 23) took 1274 ms. Trace:
I20250623 08:04:48.913020 342 tablet_bootstrap.cc:492] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.913059 952 rpcz_store.cc:276] 0623 08:04:47.638591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.682695 (+ 44104us) service_pool.cc:225] Handling call
0623 08:04:48.912665 (+1229970us) ts_tablet_manager.cc:1145] Deleting tablet d5f82dc198af45f598bf05c05a4a8090
0623 08:04:48.912673 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.912932 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1180694}
I20250623 08:04:48.913506 952 tablet_replica.cc:331] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:48.913753 952 raft_consensus.cc:2241] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:48.913950 952 raft_consensus.cc:2270] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.914455 952 ts_tablet_manager.cc:1905] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.915643 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:48.915947 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:48.916218 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:48.916393 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:48.917179 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:48.917470 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:48.917517 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:48.917691 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5af1dea3c6ca4f498ce6c039d72a14b7 (table default.Table_4b49e65313cd47e5b682a79f0394d475 [id=ad7210d2665f463c8591e79cf62d15dc]). Expected version 2 got 0
I20250623 08:04:48.918018 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:48.918560 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:48.911654 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet eef663ffdaf141aea5e6269b6b2e1c89 (table default.Table_dbd34051348e4440be23ffa0df7b19cb [id=4a301c0fc80a460bb27e15651ca0297d]) successfully deleted
W20250623 08:04:48.919201 1213 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet f20b23fd6e5c44f58b1fb2bf617c5704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f20b23fd6e5c44f58b1fb2bf617c5704 already in progress: creating tablet
W20250623 08:04:48.919427 1213 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet d5f82dc198af45f598bf05c05a4a8090 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5f82dc198af45f598bf05c05a4a8090 already in progress: creating tablet
I20250623 08:04:48.914666 342 tablet_bootstrap.cc:654] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.942409 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:48.943348 1248 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:48.947819 343 tablet_bootstrap.cc:492] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:48.947964 343 ts_tablet_manager.cc:1397] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.214s user 0.003s sys 0.002s
I20250623 08:04:48.948726 343 raft_consensus.cc:357] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.948884 343 raft_consensus.cc:383] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.948930 343 raft_consensus.cc:738] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.949229 343 consensus_queue.cc:260] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.950104 343 ts_tablet_manager.cc:1428] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:48.951627 2558 raft_consensus.cc:491] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.951726 2558 raft_consensus.cc:513] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:48.952217 2558 leader_election.cc:290] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:48.953991 32589 leader_election.cc:343] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: a8945b6f49474648ac73bcb85285680a
I20250623 08:04:48.959756 1231 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 90037c523a1f4118af268e088bb97062
I20250623 08:04:48.960258 1231 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57678 (request call id 30) took 3080 ms. Trace:
I20250623 08:04:48.960393 1231 rpcz_store.cc:276] 0623 08:04:45.880181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.736981 (+856800us) service_pool.cc:225] Handling call
0623 08:04:46.737525 (+ 544us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:46.737668 (+ 143us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:46.737742 (+ 74us) catalog_manager.cc:2799] Locking tablets
0623 08:04:46.820384 (+ 82642us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:47.341178 (+520794us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:47.341258 (+ 80us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:48.959685 (+1618427us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:48.960218 (+ 533us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.186583 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.186697 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.186716 (+ 19us) write_op.cc:435] Acquired schema lock
0623 08:04:47.186719 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:47.187266 (+ 547us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.187280 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.187285 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:47.187398 (+ 113us) tablet.cc:699] Row locks acquired
0623 08:04:47.187401 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:47.187441 (+ 40us) write_op.cc:270] Start()
0623 08:04:47.187483 (+ 42us) write_op.cc:276] Timestamp: P: 1750665887187385 usec, L: 0
0623 08:04:47.187487 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.187638 (+ 151us) log.cc:844] Serialized 725 byte log entry
0623 08:04:47.336023 (+148385us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.336599 (+ 576us) write_op.cc:301] APPLY: starting
0623 08:04:47.336611 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.336645 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.336669 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.336716 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.336750 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:47.336757 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:47.336802 (+ 45us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.336837 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.336848 (+ 11us) write_op.cc:454] Released schema lock
0623 08:04:47.336853 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.336906 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.336909 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":545,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":301,"num_ops":2,"prepare.queue_time_us":365565,"prepare.run_cpu_time_us":799,"prepare.run_wall_time_us":1157,"raft.queue_time_us":1115,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":148514}]]}
I20250623 08:04:48.966863 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 90037c523a1f4118af268e088bb97062 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:48.967173 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 90037c523a1f4118af268e088bb97062 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:48.967535 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 90037c523a1f4118af268e088bb97062 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:48.967772 2660 tablet_replica.cc:331] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:48.967958 2660 raft_consensus.cc:2241] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:48.968108 2660 raft_consensus.cc:2270] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:48.968636 2660 ts_tablet_manager.cc:1905] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:48.975282 2660 ts_tablet_manager.cc:1918] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:48.975430 2660 log.cc:1199] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/90037c523a1f4118af268e088bb97062
I20250623 08:04:48.975844 2660 ts_tablet_manager.cc:1939] T 90037c523a1f4118af268e088bb97062 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:48.977594 1213 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 90037c523a1f4118af268e088bb97062 (table default.Table_5cf87169ed0d4987b396195d518aa055 [id=5edc67ced51e4cde9dbfcf448d24b87d]) successfully deleted
I20250623 08:04:48.978447 1359 raft_consensus.cc:491] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.978587 1359 raft_consensus.cc:513] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.979369 1359 leader_election.cc:290] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:48.981945 952 ts_tablet_manager.cc:1918] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:48.987030 1230 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda] alter complete (version 1)
I20250623 08:04:48.989445 342 tablet_bootstrap.cc:492] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.989630 342 ts_tablet_manager.cc:1397] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.012s
I20250623 08:04:48.990336 2446 raft_consensus.cc:491] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:48.990386 342 raft_consensus.cc:357] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.990530 342 raft_consensus.cc:383] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.990461 2446 raft_consensus.cc:513] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.990577 342 raft_consensus.cc:738] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.990769 342 consensus_queue.cc:260] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.991043 2446 leader_election.cc:290] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:48.991231 342 ts_tablet_manager.cc:1428] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:48.991508 342 tablet_bootstrap.cc:492] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.991828 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "297bdf6c0e424190a35b55cbeefe7161" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:48.991954 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "297bdf6c0e424190a35b55cbeefe7161" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:48.992183 32647 raft_consensus.cc:2391] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:48.993093 342 tablet_bootstrap.cc:654] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:48.993315 32314 leader_election.cc:304] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:48.993938 2446 raft_consensus.cc:3058] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:48.994135 952 log.cc:1199] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/8911308855cb4414be426dbcec592f4a
I20250623 08:04:48.994428 952 ts_tablet_manager.cc:1939] T 8911308855cb4414be426dbcec592f4a P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:48.995015 342 tablet_bootstrap.cc:492] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:48.995092 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 26) took 1191 ms. Trace:
I20250623 08:04:48.995124 342 ts_tablet_manager.cc:1397] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250623 08:04:48.995165 952 rpcz_store.cc:276] 0623 08:04:47.803409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.804915 (+ 1506us) service_pool.cc:225] Handling call
0623 08:04:48.913464 (+1108549us) ts_tablet_manager.cc:1145] Deleting tablet 8911308855cb4414be426dbcec592f4a
0623 08:04:48.913473 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:48.948284 (+ 34811us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.981851 (+ 33567us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:48.995074 (+ 13223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106252}
I20250623 08:04:48.995807 342 raft_consensus.cc:357] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.995946 342 raft_consensus.cc:383] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:48.995997 342 raft_consensus.cc:738] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:48.996179 342 consensus_queue.cc:260] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:48.996577 2446 raft_consensus.cc:2747] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:48.996663 342 ts_tablet_manager.cc:1428] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:48.996817 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 8911308855cb4414be426dbcec592f4a (table default.Table_d8bf0b849fe24798821c0c3ebe8745ba [id=560402a0002c40b682be582bd02abb22]) successfully deleted
I20250623 08:04:48.996912 342 tablet_bootstrap.cc:492] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:48.998365 342 tablet_bootstrap.cc:654] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.002163 343 tablet_bootstrap.cc:492] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:49.002888 952 tablet_replica.cc:331] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:49.003080 952 raft_consensus.cc:2241] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:49.003228 952 raft_consensus.cc:2270] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:49.003712 952 ts_tablet_manager.cc:1905] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:49.004340 343 tablet_bootstrap.cc:654] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.008267 342 tablet_bootstrap.cc:492] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:49.008548 342 ts_tablet_manager.cc:1397] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20250623 08:04:49.009438 342 raft_consensus.cc:357] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.009642 342 raft_consensus.cc:383] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.009702 342 raft_consensus.cc:738] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.009917 342 consensus_queue.cc:260] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.010419 342 ts_tablet_manager.cc:1428] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:49.010738 342 tablet_bootstrap.cc:492] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:49.010759 2446 raft_consensus.cc:491] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.010982 2446 raft_consensus.cc:513] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.011866 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baccec88aefb4056be9578bca13f1591" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.012671 2446 leader_election.cc:290] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.012974 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baccec88aefb4056be9578bca13f1591" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:49.013106 32453 leader_election.cc:343] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.013463 32314 leader_election.cc:304] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:49.013865 2446 raft_consensus.cc:2747] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.015734 2659 tablet_replica.cc:331] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:49.015913 2659 raft_consensus.cc:2241] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:49.016168 2659 raft_consensus.cc:2270] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:49.016659 2659 ts_tablet_manager.cc:1905] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:49.016852 342 tablet_bootstrap.cc:654] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:49.021484 385 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.31.69.62:39465 timed out after 0.564s (SENT) [suppressed 1 similar messages]
I20250623 08:04:49.023540 2659 ts_tablet_manager.cc:1918] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:49.024160 2659 log.cc:1199] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/90037c523a1f4118af268e088bb97062
I20250623 08:04:49.024731 2659 ts_tablet_manager.cc:1939] T 90037c523a1f4118af268e088bb97062 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:49.026077 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 90037c523a1f4118af268e088bb97062 (table default.Table_5cf87169ed0d4987b396195d518aa055 [id=5edc67ced51e4cde9dbfcf448d24b87d]) successfully deleted
I20250623 08:04:49.032814 32379 raft_consensus.cc:3058] T f5f58f55070344ff98a3808f70e0c755 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.036315 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:49.048619 2446 raft_consensus.cc:491] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.049024 2446 raft_consensus.cc:513] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.049305 2439 raft_consensus.cc:491] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.049429 2439 raft_consensus.cc:513] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.049937 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a24019b797467e857b744d701fb1ed" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.049957 2439 leader_election.cc:290] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.051012 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a24019b797467e857b744d701fb1ed" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.051247 32647 raft_consensus.cc:2391] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:49.051728 32314 leader_election.cc:304] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:49.052039 2691 raft_consensus.cc:3058] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.052479 952 ts_tablet_manager.cc:1918] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:49.052592 952 log.cc:1199] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/1f76ef24c69f4b17a373b242d71da8b9
I20250623 08:04:49.053025 952 ts_tablet_manager.cc:1939] T 1f76ef24c69f4b17a373b242d71da8b9 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:49.054368 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 1f76ef24c69f4b17a373b242d71da8b9 (table default.Table_ff8fd153c41246b482ea6b6e3f24c02f [id=0270f8e6b09448b18445795985da923d]) successfully deleted
I20250623 08:04:49.054922 2691 raft_consensus.cc:2747] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.052040 2446 leader_election.cc:290] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.056545 2613 log.cc:826] T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Log is configured to *not* fsync() on all Append() calls
I20250623 08:04:49.059358 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:49.059888 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:49.060432 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:49.061174 1227 catalog_manager.cc:5582] T 0cca63991d5b4ef5bd8a9181d127ebe3 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:49.062054 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:49.062405 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
I20250623 08:04:49.063230 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:49.063848 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:49.065104 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:49.065479 952 tablet_replica.cc:331] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:49.065924 952 raft_consensus.cc:2241] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:49.067057 1227 catalog_manager.cc:5582] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:49.067417 952 raft_consensus.cc:2270] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:49.066733 32453 leader_election.cc:304] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.068462 952 ts_tablet_manager.cc:1905] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:49.068647 1359 raft_consensus.cc:2802] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:49.071244 1359 raft_consensus.cc:695] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:49.071612 1359 consensus_queue.cc:237] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.077165 2439 consensus_queue.cc:1035] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:49.079070 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:49.079542 1227 catalog_manager.cc:5582] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:49.084897 339 tablet_bootstrap.cc:492] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.085220 339 ts_tablet_manager.cc:1397] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.183s user 0.050s sys 0.059s
I20250623 08:04:49.085950 339 raft_consensus.cc:357] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.086174 339 raft_consensus.cc:383] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.086261 339 raft_consensus.cc:738] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.086522 339 consensus_queue.cc:260] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.087416 339 ts_tablet_manager.cc:1428] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:49.088675 343 tablet_bootstrap.cc:492] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.088930 343 ts_tablet_manager.cc:1397] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.002s
I20250623 08:04:49.090534 2439 raft_consensus.cc:491] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.090651 2439 raft_consensus.cc:513] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.080245 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:49.091221 2439 leader_election.cc:290] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.091452 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:49.091723 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:49.092271 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:49.092621 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:49.093040 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
W20250623 08:04:49.095826 385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 74215.5us
I20250623 08:04:49.108920 339 tablet_bootstrap.cc:492] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.110385 952 ts_tablet_manager.cc:1918] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:49.111404 952 log.cc:1199] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/90037c523a1f4118af268e088bb97062
I20250623 08:04:49.113093 2558 raft_consensus.cc:491] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.113286 2558 raft_consensus.cc:513] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.114035 2558 leader_election.cc:290] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.115128 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f8f8a132e6245f9a04b5bb6f2253c76" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:49.116197 32589 leader_election.cc:343] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.122738 2558 raft_consensus.cc:491] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.122788 339 tablet_bootstrap.cc:654] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.122941 2558 raft_consensus.cc:513] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.123656 2558 leader_election.cc:290] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.124598 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a07188a4d340bc9caf010d5202e34d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:49.125283 32589 leader_election.cc:343] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.129526 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:49.131181 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f8f8a132e6245f9a04b5bb6f2253c76" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.131335 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a07188a4d340bc9caf010d5202e34d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.133641 339 tablet_bootstrap.cc:492] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.136646 339 ts_tablet_manager.cc:1397] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20250623 08:04:49.138362 339 raft_consensus.cc:357] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.138576 339 raft_consensus.cc:383] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.138626 339 raft_consensus.cc:738] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.138922 339 consensus_queue.cc:260] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.139706 339 ts_tablet_manager.cc:1428] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250623 08:04:49.140050 339 tablet_bootstrap.cc:492] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.140723 952 ts_tablet_manager.cc:1939] T 90037c523a1f4118af268e088bb97062 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:49.141510 339 tablet_bootstrap.cc:654] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.142802 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 90037c523a1f4118af268e088bb97062 (table default.Table_5cf87169ed0d4987b396195d518aa055 [id=5edc67ced51e4cde9dbfcf448d24b87d]) successfully deleted
I20250623 08:04:49.155169 343 raft_consensus.cc:357] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.155372 343 raft_consensus.cc:383] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.155431 343 raft_consensus.cc:738] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.155642 343 consensus_queue.cc:260] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.159011 343 ts_tablet_manager.cc:1428] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.070s user 0.000s sys 0.002s
I20250623 08:04:49.159307 343 tablet_bootstrap.cc:492] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:49.159883 2558 raft_consensus.cc:491] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.160023 2558 raft_consensus.cc:513] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.160578 2558 leader_election.cc:290] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.160766 343 tablet_bootstrap.cc:654] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.161422 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bb99da3b8fa4dc6aa7f11a829d3f2d2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:49.162055 32589 leader_election.cc:343] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.166957 1227 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:49.169200 32654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d79c96bd1c4ddfb4a1dd8a9d83c3fe" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.169381 32654 raft_consensus.cc:2466] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
W20250623 08:04:49.170372 32589 leader_election.cc:343] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 4b11777e0bd04e61a5bc8bb6ec4d97c3
I20250623 08:04:49.170490 32589 leader_election.cc:304] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.170919 2558 raft_consensus.cc:2747] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:49.171046 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c337a37fcb141bca87212317d750de8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.171471 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d79c96bd1c4ddfb4a1dd8a9d83c3fe" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.173449 1823 raft_consensus.cc:491] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.173622 1823 raft_consensus.cc:513] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.174140 1823 leader_election.cc:290] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.180603 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e13dc85076b34903b35310b14dc31085" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.180836 32647 raft_consensus.cc:2466] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:49.181079 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c337a37fcb141bca87212317d750de8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.181063 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15154c85de7144a9b9e2367c70f57e2e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.181216 32651 raft_consensus.cc:2466] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:49.181270 1823 raft_consensus.cc:491] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.181386 1823 raft_consensus.cc:513] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.181968 1823 leader_election.cc:290] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.182718 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e13dc85076b34903b35310b14dc31085" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.187703 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15154c85de7144a9b9e2367c70f57e2e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.187876 32651 raft_consensus.cc:2466] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:49.188066 339 tablet_bootstrap.cc:492] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.188272 339 ts_tablet_manager.cc:1397] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.008s
I20250623 08:04:49.189296 339 raft_consensus.cc:357] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.189550 339 raft_consensus.cc:383] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.189646 339 raft_consensus.cc:738] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.189949 339 consensus_queue.cc:260] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.190702 339 ts_tablet_manager.cc:1428] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:49.191555 342 tablet_bootstrap.cc:492] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:49.191702 342 ts_tablet_manager.cc:1397] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.181s user 0.036s sys 0.068s
I20250623 08:04:49.192363 339 tablet_bootstrap.cc:492] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.192433 342 raft_consensus.cc:357] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.192598 342 raft_consensus.cc:383] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.192659 342 raft_consensus.cc:738] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.192860 342 consensus_queue.cc:260] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.193316 342 ts_tablet_manager.cc:1428] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:49.193596 342 tablet_bootstrap.cc:492] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:49.193856 32589 leader_election.cc:343] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: a8945b6f49474648ac73bcb85285680a
I20250623 08:04:49.193980 32589 leader_election.cc:304] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.194432 2446 raft_consensus.cc:491] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.194552 2446 raft_consensus.cc:513] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.195138 2446 leader_election.cc:290] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.195201 342 tablet_bootstrap.cc:654] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:49.196584 32314 leader_election.cc:343] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 8911308855cb4414be426dbcec592f4a
W20250623 08:04:49.197214 32314 leader_election.cc:343] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 8911308855cb4414be426dbcec592f4a
I20250623 08:04:49.197306 32314 leader_election.cc:304] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:49.197860 2446 raft_consensus.cc:2747] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.198431 2517 raft_consensus.cc:2747] T a8945b6f49474648ac73bcb85285680a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250623 08:04:49.202157 32589 leader_election.cc:343] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.202180 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bb99da3b8fa4dc6aa7f11a829d3f2d2" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.202494 32589 leader_election.cc:304] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:49.203068 32453 leader_election.cc:343] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:49.203328 32453 leader_election.cc:343] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.203809 2337 raft_consensus.cc:2747] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:49.204465 32589 leader_election.cc:343] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.204711 32589 leader_election.cc:304] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.205677 2337 raft_consensus.cc:2747] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.207202 343 tablet_bootstrap.cc:492] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.207337 343 ts_tablet_manager.cc:1397] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.001s
I20250623 08:04:49.207978 339 tablet_bootstrap.cc:654] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.208017 343 raft_consensus.cc:357] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.208176 343 raft_consensus.cc:383] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.208238 343 raft_consensus.cc:738] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.208441 343 consensus_queue.cc:260] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.208979 343 ts_tablet_manager.cc:1428] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:49.209249 343 tablet_bootstrap.cc:492] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:49.210641 343 tablet_bootstrap.cc:654] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.219484 2558 raft_consensus.cc:491] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.219663 2558 raft_consensus.cc:513] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.220315 2558 leader_election.cc:290] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.223639 339 tablet_bootstrap.cc:492] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.223733 342 tablet_bootstrap.cc:492] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:49.223871 342 ts_tablet_manager.cc:1397] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250623 08:04:49.223894 339 ts_tablet_manager.cc:1397] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20250623 08:04:49.224143 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba5913811e9343389c9ea0ec8b8c05ef" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.224648 342 raft_consensus.cc:357] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.224702 339 raft_consensus.cc:357] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.224825 342 raft_consensus.cc:383] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.224843 339 raft_consensus.cc:383] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.224893 339 raft_consensus.cc:738] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.224885 342 raft_consensus.cc:738] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.225106 342 consensus_queue.cc:260] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.225158 339 consensus_queue.cc:260] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.225723 342 ts_tablet_manager.cc:1428] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:49.225739 339 ts_tablet_manager.cc:1428] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:49.226009 342 tablet_bootstrap.cc:492] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:49.226125 339 tablet_bootstrap.cc:492] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.227669 342 tablet_bootstrap.cc:654] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:49.227738 32453 leader_election.cc:343] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.228330 339 tablet_bootstrap.cc:654] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.230996 1823 consensus_queue.cc:1035] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:49.232213 32453 leader_election.cc:343] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.232544 343 tablet_bootstrap.cc:492] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.232703 343 ts_tablet_manager.cc:1397] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250623 08:04:49.233019 32453 leader_election.cc:304] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.233305 1823 raft_consensus.cc:2802] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:49.233383 1823 raft_consensus.cc:491] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:49.233341 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:49.233455 1823 raft_consensus.cc:3058] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.233883 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:49.234094 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:49.234175 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:49.236294 1823 raft_consensus.cc:513] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.236897 1823 leader_election.cc:290] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.238035 342 tablet_bootstrap.cc:492] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
W20250623 08:04:49.237329 32589 leader_election.cc:343] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.238166 342 ts_tablet_manager.cc:1397] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20250623 08:04:49.238878 342 raft_consensus.cc:357] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.238912 343 raft_consensus.cc:357] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.239038 342 raft_consensus.cc:383] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.239058 343 raft_consensus.cc:383] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.239096 342 raft_consensus.cc:738] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.239110 343 raft_consensus.cc:738] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.239274 342 consensus_queue.cc:260] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.239274 343 consensus_queue.cc:260] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.239795 343 ts_tablet_manager.cc:1428] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250623 08:04:49.239817 342 ts_tablet_manager.cc:1428] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:49.240056 343 tablet_bootstrap.cc:492] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:49.240056 342 tablet_bootstrap.cc:492] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:49.241130 2446 raft_consensus.cc:491] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.241267 2446 raft_consensus.cc:513] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.241664 342 tablet_bootstrap.cc:654] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.241659 343 tablet_bootstrap.cc:654] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.241835 2446 leader_election.cc:290] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.242408 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7b9e843ff1344a1aaacf038f15f64cd" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.243021 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7b9e843ff1344a1aaacf038f15f64cd" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.243235 32647 raft_consensus.cc:2391] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:49.243584 32314 leader_election.cc:304] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:49.243988 2446 raft_consensus.cc:3058] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.246304 32368 raft_consensus.cc:3058] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.246665 2446 raft_consensus.cc:2747] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.248644 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:49.248726 339 tablet_bootstrap.cc:492] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.248843 339 ts_tablet_manager.cc:1397] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20250623 08:04:49.248873 1823 consensus_queue.cc:1035] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:49.249020 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:49.249213 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:49.249368 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:49.249626 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:49.249572 339 raft_consensus.cc:357] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.249773 339 raft_consensus.cc:383] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.249835 339 raft_consensus.cc:738] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.250029 339 consensus_queue.cc:260] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.250111 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
I20250623 08:04:49.250277 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 71fadb10e6134478b013358bba056389 (table default.Table_bb325e4f89c84cf0ae7aa94b67c08817 [id=2979e4296458439ea5753b6ba67634ac]). Expected version 1 got 0
I20250623 08:04:49.250489 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:49.250535 339 ts_tablet_manager.cc:1428] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:49.250494 32453 leader_election.cc:304] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.250818 339 tablet_bootstrap.cc:492] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.250909 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:49.252400 339 tablet_bootstrap.cc:654] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.252708 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:49.252964 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:49.253244 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:49.253505 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:49.253793 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:49.254289 1823 raft_consensus.cc:2802] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:49.254585 1823 raft_consensus.cc:491] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:49.254741 1823 raft_consensus.cc:3058] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.257730 1823 raft_consensus.cc:513] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.258173 32380 raft_consensus.cc:1273] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Refusing update from remote peer 125745849172449ba4be8244d61fdcb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 08:04:49.258335 1823 leader_election.cc:290] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.258919 32453 leader_election.cc:304] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.259305 1953 raft_consensus.cc:2802] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:49.259372 1953 raft_consensus.cc:491] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:49.259424 1953 raft_consensus.cc:3058] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.261823 1953 raft_consensus.cc:513] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.262235 2558 consensus_queue.cc:1035] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:49.262354 1953 leader_election.cc:290] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.263455 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:49.263861 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:49.264281 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 51fce46407ac4e57a74ad696e85daac7 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:49.264885 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:49.265224 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:49.266494 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:49.266819 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:49.267730 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1cee3de5953244f98428747bea912f77 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]). Expected version 1 got 0
I20250623 08:04:49.268150 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ca5c694856f44201a8aa4314a4f81681 (table default.Table_682a42dbf21d43d49ad9d85b958b121c [id=4f81957f51bd40c192ad7c716cbd893f]). Expected version 1 got 0
I20250623 08:04:49.268421 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 035bb06bc7dd42f681886def9d56eb73 (table default.Table_53879d6f821f4167bf02ffbe967f73f7 [id=b3f407a7a26e4bd5ae9a719a5c4577c0]). Expected version 1 got 0
I20250623 08:04:49.268716 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:49.269161 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7f890ee0747b4df0b82bc3b396191124 (table default.Table_7ccb5edd6b824f308d349c51d926d963 [id=9f97f6deca194137a353ea221d0e3676]). Expected version 1 got 0
I20250623 08:04:49.270090 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c50655c06cd94c67a49513c39728eaa9 (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:49.270871 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:49.271201 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:49.271907 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_8f362905029746c19e0e7ce1433c594d [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 2 got 0
I20250623 08:04:49.272388 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:49.272887 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:49.273200 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1c529e826fa84a919c018887620e4023 (table default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]). Expected version 1 got 0
I20250623 08:04:49.274154 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:49.275732 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:49.276523 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:49.277618 1231 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:49.278957 342 tablet_bootstrap.cc:492] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:49.279170 342 ts_tablet_manager.cc:1397] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.000s
I20250623 08:04:49.279975 342 raft_consensus.cc:357] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.280220 342 raft_consensus.cc:383] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.280324 342 raft_consensus.cc:738] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.280330 2558 raft_consensus.cc:491] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.280476 2558 raft_consensus.cc:513] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.280575 342 consensus_queue.cc:260] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.281005 2558 leader_election.cc:290] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.281039 342 ts_tablet_manager.cc:1428] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:49.281291 342 tablet_bootstrap.cc:492] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:49.282171 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f416cdb596a4ed5a302acd07bb84b9c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.283351 1226 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57678 (request call id 32) took 3402 ms. Trace:
I20250623 08:04:49.283478 32453 leader_election.cc:304] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.283713 2439 raft_consensus.cc:2802] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:49.283776 2439 raft_consensus.cc:491] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:49.283833 2439 raft_consensus.cc:3058] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.283484 1226 rpcz_store.cc:276] 0623 08:04:45.880389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.815591 (+935202us) service_pool.cc:225] Handling call
0623 08:04:47.185931 (+370340us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23da5a6 00007f79d9225616 00007f79d9225345 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:47.202997 (+ 17066us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:47.203213 (+ 216us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:47.203376 (+ 163us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:47.203471 (+ 95us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:47.366189 (+162718us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:49.283313 (+1917124us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.205555 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.205633 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.205638 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:47.205640 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:47.205711 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.205718 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.205722 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:47.205762 (+ 40us) tablet.cc:699] Row locks acquired
0623 08:04:47.205764 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:47.205785 (+ 21us) write_op.cc:270] Start()
0623 08:04:47.205807 (+ 22us) write_op.cc:276] Timestamp: P: 1750665887205732 usec, L: 0
0623 08:04:47.205810 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.205926 (+ 116us) log.cc:844] Serialized 405 byte log entry
0623 08:04:47.362987 (+157061us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.365776 (+ 2789us) write_op.cc:301] APPLY: starting
0623 08:04:47.365789 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.365816 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.365846 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.365880 (+ 34us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.365910 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.365918 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:47.365971 (+ 53us) log.cc:844] Serialized 23 byte log entry
0623 08:04:47.366025 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.366037 (+ 12us) write_op.cc:454] Released schema lock
0623 08:04:47.366042 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.366107 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.366111 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":850996992,"child_traces":[["op",{"apply.queue_time_us":2739,"apply.run_cpu_time_us":326,"apply.run_wall_time_us":326,"num_ops":1,"prepare.queue_time_us":1648,"prepare.run_cpu_time_us":411,"prepare.run_wall_time_us":410,"replication_time_us":157151}]]}
I20250623 08:04:49.286357 2439 raft_consensus.cc:513] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.286948 2439 leader_election.cc:290] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.288911 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f2a63b5a734c8f9b7a53361590245d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.289080 32647 raft_consensus.cc:2466] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:49.292500 1229 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: default.Table_d2473e1e63c84b229243a1c676266ca6 [id=c63ada5e02d74e049460990450ab4962] alter complete (version 1)
I20250623 08:04:49.294214 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f2a63b5a734c8f9b7a53361590245d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.303676 342 tablet_bootstrap.cc:654] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:49.307576 32589 leader_election.cc:343] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.315477 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d79c96bd1c4ddfb4a1dd8a9d83c3fe" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:49.315670 32647 raft_consensus.cc:3058] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.315582 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d79c96bd1c4ddfb4a1dd8a9d83c3fe" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:49.319001 32647 raft_consensus.cc:2466] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
W20250623 08:04:49.327678 32589 leader_election.cc:343] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.327914 32589 leader_election.cc:304] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.329360 2446 raft_consensus.cc:491] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.329506 2446 raft_consensus.cc:513] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.330200 2446 leader_election.cc:290] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.329041 2718 raft_consensus.cc:2747] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.330893 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3576624314d4e85b411aab9109683f7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.332274 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e13dc85076b34903b35310b14dc31085" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:49.332492 32379 raft_consensus.cc:3058] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.337327 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:49.331445 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e13dc85076b34903b35310b14dc31085" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:49.337807 32651 raft_consensus.cc:3058] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.339767 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c337a37fcb141bca87212317d750de8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:49.339960 32649 raft_consensus.cc:3058] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.340477 32651 raft_consensus.cc:2466] T e13dc85076b34903b35310b14dc31085 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:49.332018 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3576624314d4e85b411aab9109683f7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.341089 32645 raft_consensus.cc:2391] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:49.341764 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15154c85de7144a9b9e2367c70f57e2e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:49.342003 32651 raft_consensus.cc:3058] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.344168 1957 raft_consensus.cc:491] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.344307 1957 raft_consensus.cc:513] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.344748 32567 heartbeater.cc:344] Connected to a master server at 127.31.69.62:39465
I20250623 08:04:49.344821 1957 leader_election.cc:290] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.344877 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:49.345450 32649 raft_consensus.cc:2466] T 0c337a37fcb141bca87212317d750de8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:49.339201 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c337a37fcb141bca87212317d750de8" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:49.341424 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15154c85de7144a9b9e2367c70f57e2e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:49.346021 32368 raft_consensus.cc:3058] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.347743 343 tablet_bootstrap.cc:492] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.347994 343 ts_tablet_manager.cc:1397] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.108s user 0.008s sys 0.008s
I20250623 08:04:49.348799 2517 raft_consensus.cc:491] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.348915 2517 raft_consensus.cc:513] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.348958 343 raft_consensus.cc:357] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.349150 343 raft_consensus.cc:383] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.349210 343 raft_consensus.cc:738] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.349448 343 consensus_queue.cc:260] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.349460 2517 leader_election.cc:290] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.347796 32651 raft_consensus.cc:2466] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:49.341723 32314 leader_election.cc:304] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:49.351431 2446 raft_consensus.cc:3058] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.356220 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba5913811e9343389c9ea0ec8b8c05ef" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.356361 32519 raft_consensus.cc:1273] T b97f3019b893441bae18146fef094bd3 P 59f7c614254f42009306b13337f06576 [term 2 FOLLOWER]: Refusing update from remote peer 125745849172449ba4be8244d61fdcb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250623 08:04:49.350116 343 ts_tablet_manager.cc:1428] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:49.358139 343 tablet_bootstrap.cc:492] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:49.362789 343 tablet_bootstrap.cc:654] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.368599 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2c990297f554adbb2d97f6309758940" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:49.371037 32589 leader_election.cc:343] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.374361 342 tablet_bootstrap.cc:492] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:49.374521 342 ts_tablet_manager.cc:1397] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.093s user 0.020s sys 0.035s
I20250623 08:04:49.375021 32368 raft_consensus.cc:2466] T 0c337a37fcb141bca87212317d750de8 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:49.375512 342 raft_consensus.cc:357] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.375929 342 raft_consensus.cc:383] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.375993 342 raft_consensus.cc:738] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.376191 342 consensus_queue.cc:260] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.376783 342 ts_tablet_manager.cc:1428] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:49.377116 342 tablet_bootstrap.cc:492] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:49.378965 342 tablet_bootstrap.cc:654] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.379647 2446 raft_consensus.cc:2747] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.380093 32380 raft_consensus.cc:3058] T 4e06ef06db2643779f93617e27f100dd P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.383630 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:49.386523 342 tablet_bootstrap.cc:492] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:49.386478 2718 raft_consensus.cc:491] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:49.386665 342 ts_tablet_manager.cc:1397] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250623 08:04:49.386637 2718 raft_consensus.cc:513] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.387315 2718 leader_election.cc:290] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.387387 342 raft_consensus.cc:357] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.387544 342 raft_consensus.cc:383] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.387596 342 raft_consensus.cc:738] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.387801 342 consensus_queue.cc:260] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.388221 342 ts_tablet_manager.cc:1428] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:49.388460 342 tablet_bootstrap.cc:492] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:49.389748 342 tablet_bootstrap.cc:654] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.390535 2726 tablet.cc:1722] T 11bb55088ab24b94af3e509810890247 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:49.389983 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:49.393041 32379 raft_consensus.cc:2466] T e13dc85076b34903b35310b14dc31085 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:49.394893 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92005abc61dd40ef9da8124cadc29b6a" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.396726 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:49.400470 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:49.403791 339 tablet_bootstrap.cc:492] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.404043 339 ts_tablet_manager.cc:1397] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.153s user 0.004s sys 0.016s
I20250623 08:04:49.403787 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:49.405048 339 raft_consensus.cc:357] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.405376 339 raft_consensus.cc:383] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.405568 339 raft_consensus.cc:738] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.405916 339 consensus_queue.cc:260] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.407021 339 ts_tablet_manager.cc:1428] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:49.407440 339 tablet_bootstrap.cc:492] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.408346 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f416cdb596a4ed5a302acd07bb84b9c" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.409089 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2c990297f554adbb2d97f6309758940" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.410580 339 tablet_bootstrap.cc:654] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.407399 2728 tablet.cc:1722] T 11bb55088ab24b94af3e509810890247 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:49.416884 339 tablet_bootstrap.cc:492] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.417032 339 ts_tablet_manager.cc:1397] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250623 08:04:49.417647 339 raft_consensus.cc:357] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.417816 339 raft_consensus.cc:383] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.417867 339 raft_consensus.cc:738] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.418104 339 consensus_queue.cc:260] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.418700 339 ts_tablet_manager.cc:1428] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:49.419020 339 tablet_bootstrap.cc:492] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.420326 339 tablet_bootstrap.cc:654] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.421833 342 tablet_bootstrap.cc:492] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:49.422101 342 ts_tablet_manager.cc:1397] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.016s
I20250623 08:04:49.422892 1953 raft_consensus.cc:491] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.423028 1953 raft_consensus.cc:513] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.423120 2718 raft_consensus.cc:491] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.423154 342 raft_consensus.cc:357] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.423231 2718 raft_consensus.cc:513] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.423295 342 raft_consensus.cc:383] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.423343 342 raft_consensus.cc:738] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.423528 342 consensus_queue.cc:260] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.423789 2718 leader_election.cc:290] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.423584 1953 leader_election.cc:290] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.424010 342 ts_tablet_manager.cc:1428] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:49.424346 342 tablet_bootstrap.cc:492] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:49.424508 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:49.425824 342 tablet_bootstrap.cc:654] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.432543 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1494f83d13cd49c7a804593f45f97cc6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.432607 32428 heartbeater.cc:499] Master 127.31.69.61:33313 was elected leader, sending a full tablet report...
I20250623 08:04:49.431831 1226 catalog_manager.cc:5582] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 2, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 2 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
W20250623 08:04:49.433485 32589 leader_election.cc:343] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.434603 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:49.435369 343 tablet_bootstrap.cc:492] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.435506 343 ts_tablet_manager.cc:1397] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.006s
I20250623 08:04:49.436311 343 raft_consensus.cc:357] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.436545 343 raft_consensus.cc:383] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.436641 343 raft_consensus.cc:738] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.436914 343 consensus_queue.cc:260] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.437600 343 ts_tablet_manager.cc:1428] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250623 08:04:49.437913 343 tablet_bootstrap.cc:492] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:49.438872 1953 raft_consensus.cc:491] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.439011 1953 raft_consensus.cc:513] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.439071 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e4cde2af694f07a9176313b4b93a46" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.439625 1953 leader_election.cc:290] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.439790 343 tablet_bootstrap.cc:654] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.440861 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e4cde2af694f07a9176313b4b93a46" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.441079 32647 raft_consensus.cc:2466] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:49.443938 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_8f362905029746c19e0e7ce1433c594d [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 2 got 0
I20250623 08:04:49.449826 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
W20250623 08:04:49.453015 32589 leader_election.cc:343] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.453261 32589 leader_election.cc:304] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.453092 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 11bb55088ab24b94af3e509810890247 (table default.Table_e4c31249050c4016891036cf9507b607 [id=9a7a76e61e1b43508b6f6080f5163417]). Expected version 1 got 0
I20250623 08:04:49.453600 2718 raft_consensus.cc:2747] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.454073 2718 consensus_queue.cc:1035] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:49.456277 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:49.457464 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:49.459954 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:49.461306 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:49.462631 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:49.463774 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:49.464165 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:49.466929 1245 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 9891997e026448e39943ae13082bfe03
I20250623 08:04:49.467566 1245 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57678 (request call id 28) took 3587 ms. Trace:
I20250623 08:04:49.467706 1245 rpcz_store.cc:276] 0623 08:04:45.879940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.658868 (+778928us) service_pool.cc:225] Handling call
0623 08:04:46.677747 (+ 18879us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d9306 00007f79d9223956 00007f79d9223685 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:46.678490 (+ 743us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:46.678642 (+ 152us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:46.678723 (+ 81us) catalog_manager.cc:2799] Locking tablets
0623 08:04:47.362360 (+683637us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:47.497925 (+135565us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:47.498019 (+ 94us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:49.466865 (+1968846us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:49.467488 (+ 623us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.364449 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.364572 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.364577 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:47.364581 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:47.364735 (+ 154us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.364750 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.364758 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:47.364893 (+ 135us) tablet.cc:699] Row locks acquired
0623 08:04:47.364897 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:47.364942 (+ 45us) write_op.cc:270] Start()
0623 08:04:47.364979 (+ 37us) write_op.cc:276] Timestamp: P: 1750665887364923 usec, L: 0
0623 08:04:47.364985 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.365247 (+ 262us) log.cc:844] Serialized 790 byte log entry
0623 08:04:47.494284 (+129037us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.496982 (+ 2698us) write_op.cc:301] APPLY: starting
0623 08:04:47.496999 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.497039 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.497096 (+ 57us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.497157 (+ 61us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.497186 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:47.497193 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:47.497265 (+ 72us) log.cc:844] Serialized 29 byte log entry
0623 08:04:47.497318 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.497336 (+ 18us) write_op.cc:454] Released schema lock
0623 08:04:47.497342 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.497401 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.497405 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":29882112,"child_traces":[["op",{"apply.queue_time_us":2621,"apply.run_cpu_time_us":419,"apply.run_wall_time_us":419,"num_ops":2,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":900,"prepare.run_wall_time_us":900,"raft.queue_time_us":1133,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":129262}]]}
I20250623 08:04:49.464936 1227 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: default.Table_4b49e65313cd47e5b682a79f0394d475 [id=ad7210d2665f463c8591e79cf62d15dc] alter complete (version 2)
W20250623 08:04:49.476509 1242 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 297bdf6c0e424190a35b55cbeefe7161 consensus state has no leader
W20250623 08:04:49.477265 1242 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:50329 (request call id 148) took 3534 ms (client timeout 1000 ms). Trace:
W20250623 08:04:49.477412 1242 rpcz_store.cc:269] 0623 08:04:45.942393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:46.218614 (+276221us) service_pool.cc:225] Handling call
0623 08:04:46.271438 (+ 52824us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d2b58 00007f79d921f176 00007f79d921eea5 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:49.477220 (+3205782us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.194603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.194728 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.194735 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:47.194739 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:47.194859 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.194873 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.194878 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:47.194936 (+ 58us) tablet.cc:699] Row locks acquired
0623 08:04:47.194939 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:47.194973 (+ 34us) write_op.cc:270] Start()
0623 08:04:47.195007 (+ 34us) write_op.cc:276] Timestamp: P: 1750665887194919 usec, L: 0
0623 08:04:47.195011 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.195167 (+ 156us) log.cc:844] Serialized 527 byte log entry
0623 08:04:47.360132 (+164965us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.360312 (+ 180us) write_op.cc:301] APPLY: starting
0623 08:04:47.360329 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.360365 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.360398 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.360451 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.360483 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.360491 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:47.360578 (+ 87us) log.cc:844] Serialized 23 byte log entry
0623 08:04:47.360647 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.360666 (+ 19us) write_op.cc:454] Released schema lock
0623 08:04:47.360672 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.360767 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.360772 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1112,"reported_tablets":7,"spinlock_wait_cycles":118120704,"child_traces":[["op",{"apply.queue_time_us":133,"apply.run_cpu_time_us":439,"apply.run_wall_time_us":439,"num_ops":1,"prepare.queue_time_us":364605,"prepare.run_cpu_time_us":636,"prepare.run_wall_time_us":634,"replication_time_us":165089}]]}
I20250623 08:04:49.478158 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 9891997e026448e39943ae13082bfe03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.478289 1242 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_2ecc70ddf2dc40e081fefb676d46a68d" } modify_external_catalogs: true
I20250623 08:04:49.478482 1242 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_2ecc70ddf2dc40e081fefb676d46a68d" } modify_external_catalogs: true
I20250623 08:04:49.482926 952 tablet_replica.cc:331] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:49.483145 952 raft_consensus.cc:2241] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:49.483311 952 raft_consensus.cc:2270] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:49.483947 952 ts_tablet_manager.cc:1905] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:49.488288 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet db81250d1e164a7d94affd764fc0ff0c with delete_type TABLET_DATA_DELETED (replaced by tablet 89a24019b797467e857b744d701fb1ed) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.488620 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet d19d468ce13d4108a9badc36fc503306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.488770 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet a383d10d53b04d5699dd3c4bad8878c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.489013 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet a34da529ffde4595a23fc6596b30e3e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cee3de5953244f98428747bea912f77) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.489192 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet a9ea74d20c5f4eefbcb8508ed1774026 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.489359 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 3740dd120745493688eeb4c1ac311dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.489398 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet 1be8f8557b9a4ae2917c37b8d01cedf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:42 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.489522 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet adc3cf4355c24b7395fcb22af8520346 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:41 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.489632 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.489904 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet 3136ec837bea4619ae3995ceec9b24d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:40 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.490180 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fd478c007914d2cb09d929b50322a64 with delete_type TABLET_DATA_DELETED (replaced by tablet 51fce46407ac4e57a74ad696e85daac7) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.490643 32499 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b04e38d3da44301a450fe1fb5e2ada6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f416cdb596a4ed5a302acd07bb84b9c) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.490792 32495 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 886fc0fa39c54bddb25ac01f68d95805) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.490854 32483 tablet_service.cc:1515] Processing DeleteTablet for tablet bb06a340f82047ac8ec3516edb08f4b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:45 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.490985 32486 tablet_service.cc:1515] Processing DeleteTablet for tablet b720fdea539f423ba2d2aa9f89b09ae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.491031 32484 tablet_service.cc:1515] Processing DeleteTablet for tablet 16331ba6a34942a5a1b4ff4ac7bac55e with delete_type TABLET_DATA_DELETED (replaced by tablet 4e3920ac85e54379abbcc42dd80db504) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.491142 32491 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 92005abc61dd40ef9da8124cadc29b6a) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.491199 32496 tablet_service.cc:1515] Processing DeleteTablet for tablet b61b02cf2904404f8c0ffef7d268fefc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.492393 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 9891997e026448e39943ae13082bfe03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.492558 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b04e38d3da44301a450fe1fb5e2ada6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f416cdb596a4ed5a302acd07bb84b9c) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.492611 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 886fc0fa39c54bddb25ac01f68d95805 with delete_type TABLET_DATA_DELETED (replaced by tablet ec4d50d5f9394afe97e3b43d65e6894e) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.492777 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 886fc0fa39c54bddb25ac01f68d95805) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.492818 2659 tablet_replica.cc:331] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:49.493050 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.495754 32485 tablet_service.cc:1515] Processing DeleteTablet for tablet 59015204545e436cb558fe4465a67c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:38 UTC) from {username='slave'} at 127.0.0.1:34076
W20250623 08:04:49.496634 1213 catalog_manager.cc:5007] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): alter failed for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]): Illegal state: Tablet not RUNNING: STOPPING
I20250623 08:04:49.505019 2446 raft_consensus.cc:491] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.505195 2446 raft_consensus.cc:513] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.505887 2446 leader_election.cc:290] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.512068 952 ts_tablet_manager.cc:1918] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:49.512194 952 log.cc:1199] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9891997e026448e39943ae13082bfe03
I20250623 08:04:49.512789 1242 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 7ef27bb5b1764ef09888fb0dfcae5be7
I20250623 08:04:49.513957 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:49 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.517223 1953 raft_consensus.cc:491] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.517398 1953 raft_consensus.cc:513] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.518092 1953 leader_election.cc:290] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:49.518517 1235 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a3576624314d4e85b411aab9109683f7 consensus state has no leader
I20250623 08:04:49.521340 2659 raft_consensus.cc:2241] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:49.521641 2659 raft_consensus.cc:2270] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:49.522238 2659 ts_tablet_manager.cc:1905] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:49.522433 1235 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0c337a37fcb141bca87212317d750de8 consensus state has no leader
I20250623 08:04:49.523845 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet f97f14cbaffb46d681d8b11c0213cae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.525168 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet b61b02cf2904404f8c0ffef7d268fefc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:42984
W20250623 08:04:49.525229 1213 catalog_manager.cc:5001] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): alter failed for tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20250623 08:04:49.525789 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet c74e8c697d244ac2ac8174bb1fc76e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.526204 2446 raft_consensus.cc:491] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.526345 2446 raft_consensus.cc:513] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.528442 1953 raft_consensus.cc:491] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.528585 1953 raft_consensus.cc:513] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.529098 1953 leader_election.cc:290] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.531831 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 with delete_type TABLET_DATA_DELETED (replaced by tablet a79238b6f6e74ae89c52eefb95190264) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.538604 1242 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57678 (request call id 43) took 2760 ms. Trace:
I20250623 08:04:49.538790 1242 rpcz_store.cc:276] 0623 08:04:46.778435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.478112 (+2699677us) service_pool.cc:225] Handling call
0623 08:04:49.478597 (+ 485us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:49.478728 (+ 131us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:49.496920 (+ 18192us) catalog_manager.cc:2799] Locking tablets
0623 08:04:49.497134 (+ 214us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:49.506584 (+ 9450us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:49.506675 (+ 91us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:49.506721 (+ 46us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:49.538522 (+ 31801us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.498016 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.498150 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.498158 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:49.498162 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:49.498322 (+ 160us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.498340 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.498349 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.498484 (+ 135us) tablet.cc:699] Row locks acquired
0623 08:04:49.498487 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:49.498524 (+ 37us) write_op.cc:270] Start()
0623 08:04:49.498562 (+ 38us) write_op.cc:276] Timestamp: P: 1750665889498495 usec, L: 0
0623 08:04:49.498569 (+ 7us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.498856 (+ 287us) log.cc:844] Serialized 725 byte log entry
0623 08:04:49.502609 (+ 3753us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.502776 (+ 167us) write_op.cc:301] APPLY: starting
0623 08:04:49.502797 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.502853 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.502884 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.502968 (+ 84us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.503011 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:49.503021 (+ 10us) write_op.cc:312] APPLY: finished
0623 08:04:49.503108 (+ 87us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.503174 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.503198 (+ 24us) write_op.cc:454] Released schema lock
0623 08:04:49.503206 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.503283 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.503290 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":531,"apply.run_wall_time_us":531,"num_ops":2,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":917,"prepare.run_wall_time_us":950,"raft.queue_time_us":491,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":268,"replication_time_us":4004}]]}
W20250623 08:04:49.540030 1229 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:54519 (request call id 140) took 2293 ms (client timeout 1000 ms). Trace:
I20250623 08:04:49.540135 1242 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_dbe34c2ebf43478fb748f8e4d89c083e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:49.541142 1242 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbe34c2ebf43478fb748f8e4d89c083e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:49.540345 1229 rpcz_store.cc:269] 0623 08:04:47.246221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.339184 (+ 92963us) service_pool.cc:225] Handling call
0623 08:04:47.351482 (+ 12298us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d2b58 00007f79d921f176 00007f79d921eea5 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:49.539980 (+2188498us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.745893 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.746269 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.746277 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:47.746280 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:47.746589 (+ 309us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.746599 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.746604 (+ 5us) tablet.cc:683] Acquiring locks for 13 operations
0623 08:04:47.756988 (+ 10384us) tablet.cc:699] Row locks acquired
0623 08:04:47.756992 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:47.757038 (+ 46us) write_op.cc:270] Start()
0623 08:04:47.757076 (+ 38us) write_op.cc:276] Timestamp: P: 1750665887756977 usec, L: 0
0623 08:04:47.757080 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.757310 (+ 230us) log.cc:844] Serialized 5016 byte log entry
0623 08:04:47.785040 (+ 27730us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.785668 (+ 628us) write_op.cc:301] APPLY: starting
0623 08:04:47.785676 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.785740 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.785761 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.785964 (+ 203us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.785996 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0623 08:04:47.786004 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:47.786069 (+ 65us) log.cc:844] Serialized 95 byte log entry
0623 08:04:47.786112 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.786142 (+ 30us) write_op.cc:454] Released schema lock
0623 08:04:47.786151 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.786231 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.786240 (+ 9us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:47.814243 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.814545 (+ 302us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.814554 (+ 9us) write_op.cc:435] Acquired schema lock
0623 08:04:47.814558 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:47.814694 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.814710 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.814718 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:47.814826 (+ 108us) tablet.cc:699] Row locks acquired
0623 08:04:47.814831 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:47.814866 (+ 35us) write_op.cc:270] Start()
0623 08:04:47.814904 (+ 38us) write_op.cc:276] Timestamp: P: 1750665887814802 usec, L: 0
0623 08:04:47.814910 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.815126 (+ 216us) log.cc:844] Serialized 356 byte log entry
0623 08:04:47.819244 (+ 4118us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.819431 (+ 187us) write_op.cc:301] APPLY: starting
0623 08:04:47.819446 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.819485 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.819519 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.819574 (+ 55us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.819610 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:47.819616 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:47.819686 (+ 70us) log.cc:844] Serialized 23 byte log entry
0623 08:04:47.819782 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.819804 (+ 22us) write_op.cc:454] Released schema lock
0623 08:04:47.819813 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.828258 (+ 8445us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.828270 (+ 12us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":45,"reported_tablets":50,"spinlock_wait_cycles":23679872,"child_traces":[["op",{"apply.queue_time_us":602,"apply.run_cpu_time_us":548,"apply.run_wall_time_us":548,"num_ops":13,"prepare.queue_time_us":10119,"prepare.run_cpu_time_us":1253,"prepare.run_wall_time_us":11922,"raft.queue_time_us":49,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":27946}],["op",{"apply.queue_time_us":137,"apply.run_cpu_time_us":459,"apply.run_wall_time_us":459,"num_ops":1,"prepare.queue_time_us":20051,"prepare.run_cpu_time_us":939,"prepare.run_wall_time_us":14004,"raft.queue_time_us":233,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":259,"replication_time_us":4307}]]}
I20250623 08:04:49.542426 1229 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_f33a210d2a444f62b3beb1415a1775bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:49.544380 1229 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f33a210d2a444f62b3beb1415a1775bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:49.546119 32589 leader_election.cc:343] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.546310 32589 leader_election.cc:304] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.546618 2718 raft_consensus.cc:2747] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250623 08:04:49.554154 32589 leader_election.cc:343] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 92005abc61dd40ef9da8124cadc29b6a
I20250623 08:04:49.554401 32589 leader_election.cc:304] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.554687 2718 raft_consensus.cc:2747] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:49.554988 952 ts_tablet_manager.cc:1939] T 9891997e026448e39943ae13082bfe03 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:49.556306 952 tablet_replica.cc:331] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:49.556602 952 raft_consensus.cc:2241] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:49.556846 952 raft_consensus.cc:2270] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:49.557507 952 ts_tablet_manager.cc:1905] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:49.559644 32453 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.31.69.60:39691, user_credentials={real_user=slave}} blocked reactor thread for 98649.2us
W20250623 08:04:49.562625 32453 consensus_peers.cc:487] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:49.564358 32453 leader_election.cc:343] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:49.564647 32453 leader_election.cc:343] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:49.565352 32453 leader_election.cc:343] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.566231 32453 leader_election.cc:304] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:49.566568 1953 raft_consensus.cc:2802] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:49.566783 1953 raft_consensus.cc:695] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:49.567091 1953 consensus_queue.cc:237] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.567932 2439 raft_consensus.cc:491] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.568042 2439 raft_consensus.cc:513] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.568576 2439 leader_election.cc:290] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.569571 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 9891997e026448e39943ae13082bfe03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:49.570214 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a383d10d53b04d5699dd3c4bad8878c7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.570632 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:49 UTC) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:49.571229 2446 leader_election.cc:290] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.572103 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886fc0fa39c54bddb25ac01f68d95805" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.572245 32514 raft_consensus.cc:2466] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:49.572640 32314 leader_election.cc:304] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:49.572716 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886fc0fa39c54bddb25ac01f68d95805" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.572873 32645 raft_consensus.cc:2466] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:49.573263 2446 raft_consensus.cc:2802] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:49.573340 2446 raft_consensus.cc:491] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:49.573400 2446 raft_consensus.cc:3058] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.576448 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a383d10d53b04d5699dd3c4bad8878c7" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.576849 32314 leader_election.cc:304] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:49.576009 2446 raft_consensus.cc:513] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.578948 2446 leader_election.cc:290] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.579097 2446 raft_consensus.cc:2747] T a383d10d53b04d5699dd3c4bad8878c7 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.587960 339 tablet_bootstrap.cc:492] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.588116 339 ts_tablet_manager.cc:1397] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.169s user 0.003s sys 0.008s
I20250623 08:04:49.588982 339 raft_consensus.cc:357] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.589298 339 raft_consensus.cc:383] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.589411 339 raft_consensus.cc:738] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.589723 339 consensus_queue.cc:260] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.591112 32494 tablet_service.cc:1515] Processing DeleteTablet for tablet 74c1cdc83ffd4a09a163479ac58fd405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.591197 339 ts_tablet_manager.cc:1428] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:49.591344 32500 tablet_service.cc:1515] Processing DeleteTablet for tablet c74e8c697d244ac2ac8174bb1fc76e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.591565 32497 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:49 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.591758 339 tablet_bootstrap.cc:492] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.591768 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 92005abc61dd40ef9da8124cadc29b6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.593016 2659 ts_tablet_manager.cc:1918] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:49.593192 2659 log.cc:1199] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/9891997e026448e39943ae13082bfe03
I20250623 08:04:49.593745 2659 ts_tablet_manager.cc:1939] T 9891997e026448e39943ae13082bfe03 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:49.606071 2659 tablet_replica.cc:331] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:49.606467 2659 raft_consensus.cc:2241] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:49.606871 2659 raft_consensus.cc:2270] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:49.608073 2659 ts_tablet_manager.cc:1905] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:49.611143 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 44926.7us
I20250623 08:04:49.611377 32453 leader_election.cc:304] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:49.611608 1823 raft_consensus.cc:2802] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:49.613379 339 tablet_bootstrap.cc:654] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.617825 1823 raft_consensus.cc:695] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:49.618100 1823 consensus_queue.cc:237] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:49.626780 1227 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e7b9e843ff1344a1aaacf038f15f64cd consensus state has no leader
I20250623 08:04:49.628122 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 9dbbdbebdf3b4b50bda31a94fc0de36f with delete_type TABLET_DATA_DELETED (replaced by tablet 79014324599840038cf53ca02cbc6f7e) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.628415 2446 raft_consensus.cc:491] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.628654 2446 raft_consensus.cc:513] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.629439 2446 leader_election.cc:290] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.630110 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]) successfully deleted
I20250623 08:04:49.630625 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]) successfully deleted
I20250623 08:04:49.630743 32498 tablet_service.cc:1515] Processing DeleteTablet for tablet f5f58f55070344ff98a3808f70e0c755 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.630872 339 tablet_bootstrap.cc:492] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.631000 339 ts_tablet_manager.cc:1397] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.005s
I20250623 08:04:49.633132 339 raft_consensus.cc:357] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.635350 339 raft_consensus.cc:383] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.635950 339 raft_consensus.cc:738] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.636514 339 consensus_queue.cc:260] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.638013 339 ts_tablet_manager.cc:1428] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250623 08:04:49.635634 2446 raft_consensus.cc:491] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.639474 2446 raft_consensus.cc:513] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.640089 2446 leader_election.cc:290] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.639441 339 tablet_bootstrap.cc:492] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.641224 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 59015204545e436cb558fe4465a67c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:38 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.641320 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b213c8c685c43868df1e814c79fc61f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.641438 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet d5cd316e6f4447be958611ab07ed31fc with delete_type TABLET_DATA_DELETED (replaced by tablet c50655c06cd94c67a49513c39728eaa9) from {username='slave'} at 127.0.0.1:42984
W20250623 08:04:49.642472 1248 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4eebc32fdfa44abcbd6f04a636d02c7c consensus state has no leader
I20250623 08:04:49.644012 339 tablet_bootstrap.cc:654] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:49.644594 1248 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:50329 (request call id 150) took 1505 ms (client timeout 1000 ms). Trace:
W20250623 08:04:49.644742 1248 rpcz_store.cc:269] 0623 08:04:48.138975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:48.563552 (+424577us) service_pool.cc:225] Handling call
0623 08:04:49.644541 (+1080989us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":319,"reported_tablets":68}
I20250623 08:04:49.645581 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 9891997e026448e39943ae13082bfe03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.645679 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet db81250d1e164a7d94affd764fc0ff0c with delete_type TABLET_DATA_DELETED (replaced by tablet 89a24019b797467e857b744d701fb1ed) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.645748 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet f0e24d8d890d4d1db051f172b3ec96ec with delete_type TABLET_DATA_DELETED (replaced by tablet a1cf375214c54f3491098db07faedd9f) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.645855 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet a34da529ffde4595a23fc6596b30e3e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cee3de5953244f98428747bea912f77) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.645922 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 30dfa986dfd44658a0ffb07796030520 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a0615e6f8c346aba5e5f30362bc6de4) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.646097 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet b720fdea539f423ba2d2aa9f89b09ae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.646219 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet dcfe68d4e6ae470d816ac4c494dcaa51 with delete_type TABLET_DATA_DELETED (replaced by tablet 06fe16d2aa1c4a059e91348c40410bc2) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.646273 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 90ceb0f8d5c94c4e9427042d8a35e16c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.646437 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet f5f58f55070344ff98a3808f70e0c755 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:36 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.646584 32626 tablet_service.cc:1515] Processing DeleteTablet for tablet 90037c523a1f4118af268e088bb97062 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.646631 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 3136ec837bea4619ae3995ceec9b24d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:40 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.646451 32629 tablet_service.cc:1515] Processing DeleteTablet for tablet 6c3dcdcaa2e14cb2bd02ebb8f1d0e159 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:49.647042 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 9049e3d9b39f420bb51221808409fc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:33 UTC) from {username='slave'} at 127.0.0.1:42984
W20250623 08:04:49.659031 1235 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 88f2a63b5a734c8f9b7a53361590245d consensus state has no leader
W20250623 08:04:49.659948 1235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:50329 (request call id 149) took 2700 ms (client timeout 1000 ms). Trace:
W20250623 08:04:49.660142 1235 rpcz_store.cc:269] 0623 08:04:46.959812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:47.175436 (+215624us) service_pool.cc:225] Handling call
0623 08:04:47.190633 (+ 15197us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d2b58 00007f79d921f176 00007f79d921eea5 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:49.659894 (+2469261us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:47.828376 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:47.828525 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:47.828529 (+ 4us) write_op.cc:435] Acquired schema lock
0623 08:04:47.828532 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:47.828712 (+ 180us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:47.828735 (+ 23us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:47.828740 (+ 5us) tablet.cc:683] Acquiring locks for 4 operations
0623 08:04:47.828842 (+ 102us) tablet.cc:699] Row locks acquired
0623 08:04:47.828844 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:47.828875 (+ 31us) write_op.cc:270] Start()
0623 08:04:47.828904 (+ 29us) write_op.cc:276] Timestamp: P: 1750665887828811 usec, L: 0
0623 08:04:47.828908 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:47.829114 (+ 206us) log.cc:844] Serialized 1650 byte log entry
0623 08:04:47.833015 (+ 3901us) op_driver.cc:464] REPLICATION: finished
0623 08:04:47.833195 (+ 180us) write_op.cc:301] APPLY: starting
0623 08:04:47.833215 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:47.833285 (+ 70us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:47.833323 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:47.833444 (+ 121us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:47.833488 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0623 08:04:47.833499 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:47.833595 (+ 96us) log.cc:844] Serialized 41 byte log entry
0623 08:04:47.833669 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.833696 (+ 27us) write_op.cc:454] Released schema lock
0623 08:04:47.833706 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:47.836826 (+ 3120us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:47.836831 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":56,"spinlock_wait_cycles":27732352,"child_traces":[["op",{"apply.queue_time_us":129,"apply.run_cpu_time_us":596,"apply.run_wall_time_us":596,"num_ops":4,"prepare.queue_time_us":21670,"prepare.run_cpu_time_us":876,"prepare.run_wall_time_us":8447,"raft.queue_time_us":84,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":234,"replication_time_us":4080}]]}
W20250623 08:04:49.661235 385 leader_election.cc:336] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d31835ec4ad74332a00998d431492466 (127.31.69.62:39465): Timed out: RequestConsensusVote RPC to 127.31.69.62:39465 timed out after 0.564s (SENT)
I20250623 08:04:49.661382 385 leader_election.cc:304] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6644af0f4b2a4b998282b021ec1dc678; no voters: 9752155c20ee42fbb7ff7b7d5b682444, d31835ec4ad74332a00998d431492466
I20250623 08:04:49.661747 1235 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_9504b9b15ec14b209e2ec56b5e4e92df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:49.671444 339 tablet_bootstrap.cc:492] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
W20250623 08:04:49.671695 1227 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:54519 (request call id 141) took 1414 ms (client timeout 1000 ms). Trace:
W20250623 08:04:49.671916 1227 rpcz_store.cc:269] 0623 08:04:48.257006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:48.589124 (+332118us) service_pool.cc:225] Handling call
0623 08:04:49.212432 (+623308us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.212438 (+ 6us) write_op.cc:454] Released schema lock
0623 08:04:49.671654 (+459216us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.168296 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.168450 (+ 154us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.168459 (+ 9us) write_op.cc:435] Acquired schema lock
0623 08:04:49.168463 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:49.168645 (+ 182us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.168658 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.168662 (+ 4us) tablet.cc:683] Acquiring locks for 3 operations
0623 08:04:49.168758 (+ 96us) tablet.cc:699] Row locks acquired
0623 08:04:49.168760 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:49.168797 (+ 37us) write_op.cc:270] Start()
0623 08:04:49.168826 (+ 29us) write_op.cc:276] Timestamp: P: 1750665889168749 usec, L: 0
0623 08:04:49.168830 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.169100 (+ 270us) log.cc:844] Serialized 1275 byte log entry
0623 08:04:49.174513 (+ 5413us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.174647 (+ 134us) write_op.cc:301] APPLY: starting
0623 08:04:49.174665 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.174956 (+ 291us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.174987 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.175075 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.175116 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0623 08:04:49.175124 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:49.175219 (+ 95us) log.cc:844] Serialized 35 byte log entry
0623 08:04:49.175290 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.175315 (+ 25us) write_op.cc:454] Released schema lock
0623 08:04:49.175322 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.175405 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.175409 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:49.199960 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.200098 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.200106 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:49.200109 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:49.200230 (+ 121us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.200246 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.200252 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:49.200319 (+ 67us) tablet.cc:699] Row locks acquired
0623 08:04:49.200321 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:49.200360 (+ 39us) write_op.cc:270] Start()
0623 08:04:49.200397 (+ 37us) write_op.cc:276] Timestamp: P: 1750665889200309 usec, L: 0
0623 08:04:49.200402 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.200629 (+ 227us) log.cc:844] Serialized 356 byte log entry
0623 08:04:49.204796 (+ 4167us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.204921 (+ 125us) write_op.cc:301] APPLY: starting
0623 08:04:49.204937 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.204980 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.205015 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.205069 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.205101 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:49.205109 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:49.205187 (+ 78us) log.cc:844] Serialized 23 byte log entry
0623 08:04:49.205242 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.205261 (+ 19us) write_op.cc:454] Released schema lock
0623 08:04:49.205267 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":51241,"reported_tablets":59,"child_traces":[["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":746,"apply.run_wall_time_us":745,"num_ops":3,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":930,"prepare.run_wall_time_us":931,"raft.queue_time_us":43,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":328,"replication_time_us":5648}],["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":398,"apply.run_wall_time_us":398,"num_ops":1,"prepare.queue_time_us":11998,"prepare.run_cpu_time_us":784,"prepare.run_wall_time_us":783,"raft.queue_time_us":404,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":231,"replication_time_us":4362}]]}
I20250623 08:04:49.674062 1227 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec" } new_table_name: "default.Table_012962846ea34a55be604efd6b4d286b" modify_external_catalogs: true
I20250623 08:04:49.675321 339 ts_tablet_manager.cc:1397] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.004s
I20250623 08:04:49.672137 2446 raft_consensus.cc:491] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.676527 2446 raft_consensus.cc:513] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.677263 2446 leader_election.cc:290] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.676529 339 raft_consensus.cc:357] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.678323 1229 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 45) took 2820 ms. Trace:
I20250623 08:04:49.678545 339 raft_consensus.cc:383] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.678862 339 raft_consensus.cc:738] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.679299 339 consensus_queue.cc:260] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.680437 339 ts_tablet_manager.cc:1428] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250623 08:04:49.678613 1242 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 44) took 2819 ms. Trace:
I20250623 08:04:49.681703 339 tablet_bootstrap.cc:492] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.682103 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 185c2bf7b9254947bb76b7fe0924559e with delete_type TABLET_DATA_DELETED (replaced by tablet 04fde7d1c1744c0597e43086ca4c9541) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.682905 32498 tablet_service.cc:1515] Processing DeleteTablet for tablet 9891997e026448e39943ae13082bfe03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:46 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.683121 32492 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b213c8c685c43868df1e814c79fc61f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:49.686192 339 tablet_bootstrap.cc:654] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:49.693499 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 64483.2us
W20250623 08:04:49.694495 32453 leader_election.cc:343] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.678946 1229 rpcz_store.cc:276] 0623 08:04:46.857831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.541838 (+2684007us) service_pool.cc:225] Handling call
0623 08:04:49.545284 (+ 3446us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:49.636256 (+ 90972us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:49.645461 (+ 9205us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:49.677552 (+ 32091us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:49.678270 (+ 718us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.636898 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.637016 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.637023 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:49.637027 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:49.637138 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.637157 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.637163 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.637258 (+ 95us) tablet.cc:699] Row locks acquired
0623 08:04:49.637262 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:49.637304 (+ 42us) write_op.cc:270] Start()
0623 08:04:49.637339 (+ 35us) write_op.cc:276] Timestamp: P: 1750665889637259 usec, L: 0
0623 08:04:49.637345 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.637525 (+ 180us) log.cc:844] Serialized 433 byte log entry
0623 08:04:49.641717 (+ 4192us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.641830 (+ 113us) write_op.cc:301] APPLY: starting
0623 08:04:49.641843 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.641883 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.641913 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.642007 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.642039 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:49.642045 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:49.642112 (+ 67us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.642174 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.642192 (+ 18us) write_op.cc:454] Released schema lock
0623 08:04:49.642200 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.642273 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.642277 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":434,"apply.run_wall_time_us":433,"num_ops":2,"prepare.queue_time_us":38,"prepare.run_cpu_time_us":754,"prepare.run_wall_time_us":754,"raft.queue_time_us":68,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":310,"replication_time_us":4336}]]}
I20250623 08:04:49.681613 1242 rpcz_store.cc:276] 0623 08:04:46.857701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.539894 (+2682193us) service_pool.cc:225] Handling call
0623 08:04:49.541253 (+ 1359us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:49.541763 (+ 510us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:49.634032 (+ 92269us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:49.634145 (+ 113us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:49.676741 (+ 42596us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.542814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.542950 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.542960 (+ 10us) write_op.cc:435] Acquired schema lock
0623 08:04:49.542965 (+ 5us) tablet.cc:660] Decoding operations
0623 08:04:49.543057 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.543073 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.543080 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.543160 (+ 80us) tablet.cc:699] Row locks acquired
0623 08:04:49.543163 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:49.543197 (+ 34us) write_op.cc:270] Start()
0623 08:04:49.543235 (+ 38us) write_op.cc:276] Timestamp: P: 1750665889543163 usec, L: 0
0623 08:04:49.543238 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.543426 (+ 188us) log.cc:844] Serialized 433 byte log entry
0623 08:04:49.631427 (+ 88001us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.631550 (+ 123us) write_op.cc:301] APPLY: starting
0623 08:04:49.631571 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.631627 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.631683 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.631846 (+ 163us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.631895 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:49.631905 (+ 10us) write_op.cc:312] APPLY: finished
0623 08:04:49.631998 (+ 93us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.632074 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.632100 (+ 26us) write_op.cc:454] Released schema lock
0623 08:04:49.632109 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.632198 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.632204 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":61,"apply.run_cpu_time_us":634,"apply.run_wall_time_us":633,"num_ops":2,"prepare.queue_time_us":82,"prepare.run_cpu_time_us":711,"prepare.run_wall_time_us":713,"raft.queue_time_us":71,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":220,"replication_time_us":88146}]]}
I20250623 08:04:49.696067 1242 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_1026838e4fb1446488330a74d347907f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:49.696866 1242 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1026838e4fb1446488330a74d347907f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:49.697131 1229 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_caf786518e614b1cb613c11baa90f3cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:49.697751 1229 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_caf786518e614b1cb613c11baa90f3cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:49.702965 339 tablet_bootstrap.cc:492] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.704272 339 ts_tablet_manager.cc:1397] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20250623 08:04:49.705381 339 raft_consensus.cc:357] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.707255 339 raft_consensus.cc:383] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.707610 339 raft_consensus.cc:738] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.708096 339 consensus_queue.cc:260] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.710218 339 ts_tablet_manager.cc:1428] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250623 08:04:49.711616 339 tablet_bootstrap.cc:492] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.713630 339 tablet_bootstrap.cc:654] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.716763 2749 tablet_replica.cc:331] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:49.717123 2749 raft_consensus.cc:2241] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:49.717353 2749 raft_consensus.cc:2270] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:49.717964 2749 ts_tablet_manager.cc:1905] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:49.725438 2726 tablet.cc:1722] T f217a8eb12014fcf89b9df4a2cddf5f6 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:49.727308 2769 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250623 08:04:49.729600 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 152751us
W20250623 08:04:49.730644 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 100411us
I20250623 08:04:49.734668 1242 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 48) took 2786 ms. Trace:
W20250623 08:04:49.730068 385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.62:39465, user_credentials={real_user=slave}} blocked reactor thread for 68837.7us
I20250623 08:04:49.741173 2755 raft_consensus.cc:491] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:49.741503 2755 raft_consensus.cc:513] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.743746 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf753ede17f429b82612ad12e13185c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.744069 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886fc0fa39c54bddb25ac01f68d95805" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:49.744146 32513 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:49.744220 32514 raft_consensus.cc:3058] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.744262 32513 raft_consensus.cc:2466] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:49.745040 1359 raft_consensus.cc:491] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.745151 1359 raft_consensus.cc:513] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.745630 1359 leader_election.cc:290] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.746887 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d79c96bd1c4ddfb4a1dd8a9d83c3fe" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:49.746950 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 51706.1us
I20250623 08:04:49.747076 32518 raft_consensus.cc:2391] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:49.747586 32514 raft_consensus.cc:2466] T 886fc0fa39c54bddb25ac01f68d95805 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:49.748505 2755 leader_election.cc:290] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.755940 1242 rpcz_store.cc:276] 0623 08:04:46.948537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.695836 (+2747299us) service_pool.cc:225] Handling call
0623 08:04:49.697048 (+ 1212us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:49.697644 (+ 596us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:49.734183 (+ 36539us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.734193 (+ 10us) write_op.cc:454] Released schema lock
0623 08:04:49.734319 (+ 126us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:49.734445 (+ 126us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:49.734584 (+ 139us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.698212 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.698324 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.698331 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:49.698333 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:49.698414 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.698428 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.698433 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.698504 (+ 71us) tablet.cc:699] Row locks acquired
0623 08:04:49.698506 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:49.698551 (+ 45us) write_op.cc:270] Start()
0623 08:04:49.698585 (+ 34us) write_op.cc:276] Timestamp: P: 1750665889698501 usec, L: 0
0623 08:04:49.698594 (+ 9us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.698790 (+ 196us) log.cc:844] Serialized 433 byte log entry
0623 08:04:49.710316 (+ 11526us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.710600 (+ 284us) write_op.cc:301] APPLY: starting
0623 08:04:49.710619 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.710672 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.710708 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.710888 (+ 180us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.710929 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:49.710941 (+ 12us) write_op.cc:312] APPLY: finished
0623 08:04:49.711021 (+ 80us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.711088 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.711109 (+ 21us) write_op.cc:454] Released schema lock
0623 08:04:49.711117 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":216,"apply.run_cpu_time_us":569,"apply.run_wall_time_us":576,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":679,"prepare.run_wall_time_us":679,"raft.queue_time_us":727,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":305,"replication_time_us":11693}]]}
I20250623 08:04:49.747125 32453 leader_election.cc:304] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.759210 32453 leader_election.cc:304] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.764425 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28d79c96bd1c4ddfb4a1dd8a9d83c3fe" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.764532 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bcf753ede17f429b82612ad12e13185c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.764695 32645 raft_consensus.cc:2391] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:49.764752 32651 raft_consensus.cc:2391] T bcf753ede17f429b82612ad12e13185c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:49.764452 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "886fc0fa39c54bddb25ac01f68d95805" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:49.765120 32647 raft_consensus.cc:3058] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.765533 2773 raft_consensus.cc:491] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.765667 2773 raft_consensus.cc:513] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.766364 2773 leader_election.cc:290] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.767912 32647 raft_consensus.cc:2466] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:49.768277 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.768610 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92005abc61dd40ef9da8124cadc29b6a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.768757 32651 raft_consensus.cc:2466] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:49.768791 32645 raft_consensus.cc:2466] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:49.769721 2726 tablet.cc:1722] T 74da35cd34c14284af004001b25aa531 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:49.773865 2726 tablet.cc:1722] T baccec88aefb4056be9578bca13f1591 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:49.778877 343 tablet_bootstrap.cc:492] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.779067 343 ts_tablet_manager.cc:1397] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.341s user 0.034s sys 0.046s
I20250623 08:04:49.780045 343 raft_consensus.cc:357] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.780283 343 raft_consensus.cc:383] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.780392 343 raft_consensus.cc:738] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.780637 343 consensus_queue.cc:260] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.781296 343 ts_tablet_manager.cc:1428] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:49.781605 343 tablet_bootstrap.cc:492] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:49.783924 343 tablet_bootstrap.cc:654] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.755695 2728 tablet.cc:1722] T f217a8eb12014fcf89b9df4a2cddf5f6 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:49.757846 339 tablet_bootstrap.cc:492] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.790694 339 ts_tablet_manager.cc:1397] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.079s user 0.021s sys 0.005s
I20250623 08:04:49.791498 339 raft_consensus.cc:357] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.791445 1242 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_d0add936203f42e78a69c35025573ed6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:49.791663 339 raft_consensus.cc:383] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.791715 339 raft_consensus.cc:738] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.756613 1229 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 49) took 2733 ms. Trace:
I20250623 08:04:49.791870 343 tablet_bootstrap.cc:492] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.791980 343 ts_tablet_manager.cc:1397] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20250623 08:04:49.791908 339 consensus_queue.cc:260] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:49.792065 1242 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0add936203f42e78a69c35025573ed6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:49.792375 339 ts_tablet_manager.cc:1428] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:49.791877 1229 rpcz_store.cc:276] 0623 08:04:47.023501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.696920 (+2673419us) service_pool.cc:225] Handling call
0623 08:04:49.697899 (+ 979us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:49.698310 (+ 411us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:49.748793 (+ 50483us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:49.756423 (+ 7630us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:49.756568 (+ 145us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.698937 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.699056 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.699062 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:49.699065 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:49.699152 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.699163 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.699170 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.699247 (+ 77us) tablet.cc:699] Row locks acquired
0623 08:04:49.699251 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:49.699282 (+ 31us) write_op.cc:270] Start()
0623 08:04:49.699311 (+ 29us) write_op.cc:276] Timestamp: P: 1750665889699233 usec, L: 0
0623 08:04:49.699317 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.699482 (+ 165us) log.cc:844] Serialized 433 byte log entry
0623 08:04:49.710379 (+ 10897us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.711226 (+ 847us) write_op.cc:301] APPLY: starting
0623 08:04:49.711240 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.711275 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.711301 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.711409 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.711444 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:49.711454 (+ 10us) write_op.cc:312] APPLY: finished
0623 08:04:49.711509 (+ 55us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.711562 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.711577 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:49.711584 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":796,"apply.run_cpu_time_us":416,"apply.run_wall_time_us":416,"num_ops":2,"prepare.queue_time_us":251,"prepare.run_cpu_time_us":647,"prepare.run_wall_time_us":1056,"raft.queue_time_us":397,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":11045,"thread_start_us":49189,"threads_started":1}]]}
I20250623 08:04:49.792645 339 tablet_bootstrap.cc:492] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.792609 343 raft_consensus.cc:357] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.792764 343 raft_consensus.cc:383] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.792811 343 raft_consensus.cc:738] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.793009 343 consensus_queue.cc:260] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.793105 1229 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_a65a0f1004274054826e1df3df7ff595"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:49.793532 343 ts_tablet_manager.cc:1428] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250623 08:04:49.759500 1953 raft_consensus.cc:2802] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:49.783610 342 tablet_bootstrap.cc:492] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:49.793653 1957 raft_consensus.cc:2802] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:49.793710 1953 raft_consensus.cc:491] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:49.793783 1953 raft_consensus.cc:3058] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:49.793784 1229 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a65a0f1004274054826e1df3df7ff595 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:49.793817 342 ts_tablet_manager.cc:1397] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.370s user 0.036s sys 0.072s
I20250623 08:04:49.794481 339 tablet_bootstrap.cc:654] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.794678 343 tablet_bootstrap.cc:492] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:49.763212 1235 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9504b9b15ec14b209e2ec56b5e4e92df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250623 08:04:49.795951 1213 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e7b9e843ff1344a1aaacf038f15f64cd consensus state has no leader
I20250623 08:04:49.797747 343 tablet_bootstrap.cc:654] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.802307 339 tablet_bootstrap.cc:492] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.802428 339 ts_tablet_manager.cc:1397] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20250623 08:04:49.803458 339 raft_consensus.cc:357] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.803686 339 raft_consensus.cc:383] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.803803 339 raft_consensus.cc:738] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.804062 339 consensus_queue.cc:260] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.763401 32314 leader_election.cc:304] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:49.804600 339 ts_tablet_manager.cc:1428] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:49.804657 2755 raft_consensus.cc:2802] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:49.804736 2755 raft_consensus.cc:491] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:49.804797 2755 raft_consensus.cc:3058] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.804855 339 tablet_bootstrap.cc:492] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:49.806105 339 tablet_bootstrap.cc:654] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.807415 2755 raft_consensus.cc:513] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.808048 2755 leader_election.cc:290] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.808583 1953 raft_consensus.cc:513] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:49.808928 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 45536.1us
I20250623 08:04:49.810886 32314 leader_election.cc:304] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:49.813181 342 raft_consensus.cc:357] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.815202 342 raft_consensus.cc:383] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.815276 342 raft_consensus.cc:738] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.815464 342 consensus_queue.cc:260] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.815615 343 tablet_bootstrap.cc:492] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.813539 1957 raft_consensus.cc:695] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:49.815743 343 ts_tablet_manager.cc:1397] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20250623 08:04:49.815856 1957 consensus_queue.cc:237] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.818790 343 raft_consensus.cc:357] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.818958 343 raft_consensus.cc:383] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.819017 343 raft_consensus.cc:738] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.819214 343 consensus_queue.cc:260] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.819697 343 ts_tablet_manager.cc:1428] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250623 08:04:49.819986 343 tablet_bootstrap.cc:492] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:49.821638 343 tablet_bootstrap.cc:654] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.827009 1953 leader_election.cc:290] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.827802 2755 raft_consensus.cc:2802] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:49.827996 2755 raft_consensus.cc:695] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:49.828341 2755 consensus_queue.cc:237] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.830843 2718 raft_consensus.cc:491] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.831020 2718 raft_consensus.cc:513] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.831624 2718 leader_election.cc:290] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.835323 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.835641 32374 raft_consensus.cc:2391] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
W20250623 08:04:49.838939 32314 leader_election.cc:343] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 92005abc61dd40ef9da8124cadc29b6a
I20250623 08:04:49.840526 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:49.840680 32647 raft_consensus.cc:3058] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.843389 32647 raft_consensus.cc:2466] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:49.836742 342 ts_tablet_manager.cc:1428] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20250623 08:04:49.846741 342 tablet_bootstrap.cc:492] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:49.848366 342 tablet_bootstrap.cc:654] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.849642 2749 ts_tablet_manager.cc:1918] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:49.849993 2749 log.cc:1199] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/9891997e026448e39943ae13082bfe03
I20250623 08:04:49.851284 2749 ts_tablet_manager.cc:1939] T 9891997e026448e39943ae13082bfe03 P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:49.838426 32453 leader_election.cc:304] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.839831 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:49.860010 32518 raft_consensus.cc:3058] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.862865 32518 raft_consensus.cc:2466] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:49.863796 2720 raft_consensus.cc:2802] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:49.863977 2720 raft_consensus.cc:695] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:49.864259 2720 consensus_queue.cc:237] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.864475 2776 consensus_queue.cc:1035] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:49.864579 2659 ts_tablet_manager.cc:1918] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:49.864698 2659 log.cc:1199] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/6b04e38d3da44301a450fe1fb5e2ada6
I20250623 08:04:49.865088 2659 ts_tablet_manager.cc:1939] T 6b04e38d3da44301a450fe1fb5e2ada6 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:49.866328 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 6b04e38d3da44301a450fe1fb5e2ada6 (table default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e]) successfully deleted
I20250623 08:04:49.867107 2659 tablet_replica.cc:331] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:49.867959 2659 raft_consensus.cc:2241] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:49.868463 2659 raft_consensus.cc:2270] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:49.768494 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:49.871387 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:49.871831 2773 raft_consensus.cc:491] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.871963 2773 raft_consensus.cc:513] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.872026 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:49.872354 2728 tablet.cc:1722] T baccec88aefb4056be9578bca13f1591 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:49.872510 2773 leader_election.cc:290] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.872602 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:49.872866 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:49.873386 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:49.874226 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:49.874727 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:49.877354 2728 tablet.cc:1722] T 74da35cd34c14284af004001b25aa531 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:49.875486 1245 catalog_manager.cc:5582] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:49.880324 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:49.881438 1957 raft_consensus.cc:491] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.881661 1957 raft_consensus.cc:513] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.882432 1957 leader_election.cc:290] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.887420 2718 raft_consensus.cc:491] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.887634 2718 raft_consensus.cc:513] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.888351 2718 leader_election.cc:290] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.888845 32314 leader_election.cc:304] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:49.889374 32314 leader_election.cc:304] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:49.889693 2446 raft_consensus.cc:3058] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.889954 32314 leader_election.cc:304] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:49.890244 2755 raft_consensus.cc:3058] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.892277 2446 raft_consensus.cc:2747] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.892414 2446 raft_consensus.cc:2802] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250623 08:04:49.892472 2446 raft_consensus.cc:491] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:49.892544 2446 raft_consensus.cc:3058] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:49.892621 2755 raft_consensus.cc:2747] T bcf753ede17f429b82612ad12e13185c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:49.895246 2446 raft_consensus.cc:513] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.895808 2446 leader_election.cc:290] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Requested vote from peers 59f7c614254f42009306b13337f06576 (127.31.69.2:40197), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.911478 1359 raft_consensus.cc:491] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.911641 1359 raft_consensus.cc:513] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.912178 1359 leader_election.cc:290] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.912673 2659 ts_tablet_manager.cc:1905] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:49.914783 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 51017.9us
I20250623 08:04:49.915378 2776 consensus_queue.cc:1035] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:49.915725 1823 consensus_queue.cc:1035] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:49.918885 1953 raft_consensus.cc:491] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.919219 1953 raft_consensus.cc:513] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.920472 1953 leader_election.cc:290] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.923978 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ae0a411a3e54c45b8d5c5b9b0eea762" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.930101 2776 consensus_queue.cc:1035] T 185c2bf7b9254947bb76b7fe0924559e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:49.930195 32453 leader_election.cc:304] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:49.930826 2776 raft_consensus.cc:2802] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:49.930922 2776 raft_consensus.cc:491] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:49.931018 2776 raft_consensus.cc:3058] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:49.932008 343 tablet_bootstrap.cc:492] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.932134 343 ts_tablet_manager.cc:1397] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.038s
I20250623 08:04:49.933487 1213 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 9891997e026448e39943ae13082bfe03 (table default.Table_811a9724f3234933a3f6352f67b17e73 [id=a595e975752c4d63b7b2726a8de5eb99]) successfully deleted
W20250623 08:04:49.934039 1213 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ef27bb5b1764ef09888fb0dfcae5be7 already in progress: creating tablet
I20250623 08:04:49.934137 2776 raft_consensus.cc:513] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.935237 2776 leader_election.cc:290] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.936280 343 raft_consensus.cc:357] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.936446 343 raft_consensus.cc:383] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.936503 343 raft_consensus.cc:738] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.936753 343 consensus_queue.cc:260] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:49.937124 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 47178.1us
I20250623 08:04:49.938175 1235 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 46) took 3076 ms. Trace:
I20250623 08:04:49.938337 1235 rpcz_store.cc:276] 0623 08:04:46.861723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.661520 (+2799797us) service_pool.cc:225] Handling call
0623 08:04:49.794810 (+133290us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:49.795204 (+ 394us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:49.910001 (+114797us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:49.937973 (+ 27972us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:49.938117 (+ 144us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.899154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.899258 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.899265 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:49.899269 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:49.899353 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.899363 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.899370 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.899454 (+ 84us) tablet.cc:699] Row locks acquired
0623 08:04:49.899458 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:49.899487 (+ 29us) write_op.cc:270] Start()
0623 08:04:49.899513 (+ 26us) write_op.cc:276] Timestamp: P: 1750665889899417 usec, L: 0
0623 08:04:49.899517 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.899664 (+ 147us) log.cc:844] Serialized 433 byte log entry
0623 08:04:49.908585 (+ 8921us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.909223 (+ 638us) write_op.cc:301] APPLY: starting
0623 08:04:49.909232 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.909260 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.909280 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.909380 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.909406 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:49.909414 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:49.909463 (+ 49us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.909504 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.909517 (+ 13us) write_op.cc:454] Released schema lock
0623 08:04:49.909535 (+ 18us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.909593 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.909596 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":603,"apply.run_cpu_time_us":365,"apply.run_wall_time_us":364,"mutex_wait_us":102721,"num_ops":2,"prepare.queue_time_us":103571,"prepare.run_cpu_time_us":564,"prepare.run_wall_time_us":563,"replication_time_us":9037}]]}
I20250623 08:04:49.939684 1235 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_5e73ae1fb8be4c9da3c5b9b75af18187"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:49.940037 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
W20250623 08:04:49.940451 1235 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e73ae1fb8be4c9da3c5b9b75af18187 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:49.941779 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]). Expected version 2 got 0
I20250623 08:04:49.943413 1229 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 51) took 2740 ms. Trace:
I20250623 08:04:49.943580 1229 rpcz_store.cc:276] 0623 08:04:47.202743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.792942 (+2590199us) service_pool.cc:225] Handling call
0623 08:04:49.793874 (+ 932us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:49.794323 (+ 449us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:49.910560 (+116237us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:49.943260 (+ 32700us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:49.943357 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.898343 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.898465 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.898471 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:49.898476 (+ 5us) tablet.cc:660] Decoding operations
0623 08:04:49.898591 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.898603 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.898611 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.898712 (+ 101us) tablet.cc:699] Row locks acquired
0623 08:04:49.898730 (+ 18us) write_op.cc:260] PREPARE: finished
0623 08:04:49.898760 (+ 30us) write_op.cc:270] Start()
0623 08:04:49.898809 (+ 49us) write_op.cc:276] Timestamp: P: 1750665889898691 usec, L: 0
0623 08:04:49.898814 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.899018 (+ 204us) log.cc:844] Serialized 433 byte log entry
0623 08:04:49.908498 (+ 9480us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.908683 (+ 185us) write_op.cc:301] APPLY: starting
0623 08:04:49.908697 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.908751 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.908784 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.908892 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.908932 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:49.908943 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:49.909010 (+ 67us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.909063 (+ 53us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.909080 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:49.909086 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.909147 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.909150 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":133,"apply.run_cpu_time_us":459,"apply.run_wall_time_us":459,"mutex_wait_us":101633,"num_ops":2,"prepare.queue_time_us":103616,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":748,"replication_time_us":9656}]]}
I20250623 08:04:49.944517 1229 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_6220ff02a44c42e888ef49108c79ec0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250623 08:04:49.945197 1229 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6220ff02a44c42e888ef49108c79ec0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:49.947937 1242 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 50) took 2924 ms. Trace:
I20250623 08:04:49.948077 1242 rpcz_store.cc:276] 0623 08:04:47.023638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.791192 (+2767554us) service_pool.cc:225] Handling call
0623 08:04:49.792162 (+ 970us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:49.792624 (+ 462us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:49.914153 (+121529us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:49.947792 (+ 33639us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:49.947883 (+ 91us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.896371 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.896477 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.896483 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:49.896485 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:49.896597 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.896608 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.896613 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.896680 (+ 67us) tablet.cc:699] Row locks acquired
0623 08:04:49.896682 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:49.896718 (+ 36us) write_op.cc:270] Start()
0623 08:04:49.896762 (+ 44us) write_op.cc:276] Timestamp: P: 1750665889896647 usec, L: 0
0623 08:04:49.896766 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.896955 (+ 189us) log.cc:844] Serialized 433 byte log entry
0623 08:04:49.902658 (+ 5703us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.907079 (+ 4421us) write_op.cc:301] APPLY: starting
0623 08:04:49.907099 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.907155 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.907192 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.907304 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.907345 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:49.907356 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:49.907439 (+ 83us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.907687 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.907708 (+ 21us) write_op.cc:454] Released schema lock
0623 08:04:49.907715 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.907787 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.907795 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4348,"apply.run_cpu_time_us":526,"apply.run_wall_time_us":707,"mutex_wait_us":1228,"num_ops":2,"prepare.queue_time_us":103253,"prepare.run_cpu_time_us":699,"prepare.run_wall_time_us":700,"raft.queue_time_us":478,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":232,"replication_time_us":5859}]]}
I20250623 08:04:49.943434 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:49.958681 1245 catalog_manager.cc:5582] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:49.968981 1235 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 52) took 2588 ms. Trace:
I20250623 08:04:49.969086 1235 rpcz_store.cc:276] 0623 08:04:47.380751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.939447 (+2558696us) service_pool.cc:225] Handling call
0623 08:04:49.940659 (+ 1212us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:49.941147 (+ 488us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:49.968650 (+ 27503us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:49.968822 (+ 172us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:49.968925 (+ 103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.942155 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.942261 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.942268 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:49.942272 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:49.942358 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.942370 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.942375 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.942467 (+ 92us) tablet.cc:699] Row locks acquired
0623 08:04:49.942470 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:49.942504 (+ 34us) write_op.cc:270] Start()
0623 08:04:49.942543 (+ 39us) write_op.cc:276] Timestamp: P: 1750665889942480 usec, L: 0
0623 08:04:49.942547 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.942729 (+ 182us) log.cc:844] Serialized 433 byte log entry
0623 08:04:49.965592 (+ 22863us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.965749 (+ 157us) write_op.cc:301] APPLY: starting
0623 08:04:49.965765 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.965826 (+ 61us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.965872 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.965980 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.966014 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:49.966023 (+ 9us) write_op.cc:312] APPLY: finished
0623 08:04:49.966098 (+ 75us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.966164 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.966182 (+ 18us) write_op.cc:454] Released schema lock
0623 08:04:49.966189 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.966267 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.966273 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":504,"apply.run_wall_time_us":504,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":712,"prepare.run_wall_time_us":1038,"raft.queue_time_us":3911,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":265,"replication_time_us":23008}]]}
I20250623 08:04:49.969734 1235 master_service.cc:948] ReplaceTablet: received request to replace tablet 2b405f87bb1e4d0686a1f627835d901f from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:49.972313 1229 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 53) took 2586 ms. Trace:
I20250623 08:04:49.972432 1229 rpcz_store.cc:276] 0623 08:04:47.385721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.944277 (+2558556us) service_pool.cc:225] Handling call
0623 08:04:49.945348 (+ 1071us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:49.945887 (+ 539us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:49.972077 (+ 26190us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:49.972186 (+ 109us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:49.972277 (+ 91us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.946434 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.946546 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.946552 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:49.946555 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:49.946632 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.946639 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.946645 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.946703 (+ 58us) tablet.cc:699] Row locks acquired
0623 08:04:49.946705 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:49.946732 (+ 27us) write_op.cc:270] Start()
0623 08:04:49.946781 (+ 49us) write_op.cc:276] Timestamp: P: 1750665889946709 usec, L: 0
0623 08:04:49.946787 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.946934 (+ 147us) log.cc:844] Serialized 433 byte log entry
0623 08:04:49.965657 (+ 18723us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.966339 (+ 682us) write_op.cc:301] APPLY: starting
0623 08:04:49.966349 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.966384 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.966413 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.966492 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.966534 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:49.966540 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:49.966599 (+ 59us) log.cc:844] Serialized 29 byte log entry
0623 08:04:49.966636 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.966654 (+ 18us) write_op.cc:454] Released schema lock
0623 08:04:49.966660 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.967124 (+ 464us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.967131 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":651,"apply.run_cpu_time_us":380,"apply.run_wall_time_us":379,"num_ops":2,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":585,"prepare.run_wall_time_us":586,"raft.queue_time_us":357,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":18855}]]}
I20250623 08:04:49.973256 1229 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_d281efc20a4542faa2838570433fd540" } new_table_name: "default.Table_523e0491c028492db09a84adbbfe5baa" modify_external_catalogs: true
I20250623 08:04:49.977458 1242 master_service.cc:948] ReplaceTablet: received request to replace tablet 1c529e826fa84a919c018887620e4023 from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:49.979537 343 ts_tablet_manager.cc:1428] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.047s user 0.001s sys 0.001s
I20250623 08:04:49.979851 343 tablet_bootstrap.cc:492] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:49.981473 343 tablet_bootstrap.cc:654] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.982230 339 tablet_bootstrap.cc:492] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:49.982384 339 ts_tablet_manager.cc:1397] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.178s user 0.020s sys 0.083s
I20250623 08:04:49.982708 2775 raft_consensus.cc:491] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.983246 339 raft_consensus.cc:357] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.983307 2775 raft_consensus.cc:513] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.983440 339 raft_consensus.cc:383] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:49.983497 339 raft_consensus.cc:738] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:49.983736 339 consensus_queue.cc:260] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.983875 2775 leader_election.cc:290] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.983073 2774 raft_consensus.cc:491] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:49.984076 2774 raft_consensus.cc:513] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:49.984275 339 ts_tablet_manager.cc:1428] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:49.984571 2774 leader_election.cc:290] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:49.984589 339 tablet_bootstrap.cc:492] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:49.987797 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 57651us
I20250623 08:04:49.989359 339 tablet_bootstrap.cc:654] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:49.990931 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b8cfcbce6a472ab2daae28226cf766" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.991156 32647 raft_consensus.cc:2466] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:49.991693 2777 tablet.cc:1722] T 5f957f5575e9475eb9f639e3ad5cb809 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250623 08:04:49.992903 32589 leader_election.cc:343] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:49.991703 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f080a682e532406a83712b6f3e6d2f7b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:49.993827 32647 raft_consensus.cc:2466] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:49.993884 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.994201 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bfad68d1f20486997ea1b0a8d97206b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.994225 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:49.994462 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.3:35225, user_credentials={real_user=slave}} blocked reactor thread for 56739.3us
I20250623 08:04:49.995122 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cee3de5953244f98428747bea912f77" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:49.995289 2728 tablet.cc:1722] T 5f957f5575e9475eb9f639e3ad5cb809 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:49.996971 32314 leader_election.cc:304] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:49.997592 2446 raft_consensus.cc:2802] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:49.997704 2446 raft_consensus.cc:695] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:49.997917 2446 consensus_queue.cc:237] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:49.998294 343 tablet_bootstrap.cc:492] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:49.998400 343 ts_tablet_manager.cc:1397] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.005s
I20250623 08:04:50.000923 952 ts_tablet_manager.cc:1918] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:50.001111 952 log.cc:1199] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/db81250d1e164a7d94affd764fc0ff0c
I20250623 08:04:50.002951 32647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92005abc61dd40ef9da8124cadc29b6a" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.003106 32647 raft_consensus.cc:3058] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Advancing to term 2
I20250623 08:04:50.005642 32647 raft_consensus.cc:2466] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 2.
I20250623 08:04:50.006901 32649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.007076 32649 raft_consensus.cc:2466] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.009590 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.009840 32646 raft_consensus.cc:2391] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:50.014570 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "492ffaeaaa314dcfb7512d8f23946a67" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.014803 32648 raft_consensus.cc:2466] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.014981 32646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30dfa986dfd44658a0ffb07796030520" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.015127 32646 raft_consensus.cc:2466] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.018887 32652 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f2a63b5a734c8f9b7a53361590245d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.019030 32652 raft_consensus.cc:3058] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.021703 32652 raft_consensus.cc:2466] T 88f2a63b5a734c8f9b7a53361590245d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.021953 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eebc32fdfa44abcbd6f04a636d02c7c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.022099 32653 raft_consensus.cc:2466] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.023166 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcfe68d4e6ae470d816ac4c494dcaa51" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.023314 32648 raft_consensus.cc:2466] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.024530 32640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e4cde2af694f07a9176313b4b93a46" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.024739 32640 raft_consensus.cc:3058] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.026981 32643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a07188a4d340bc9caf010d5202e34d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.027164 32643 raft_consensus.cc:2466] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.036478 32640 raft_consensus.cc:2466] T c8e4cde2af694f07a9176313b4b93a46 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.049551 2720 raft_consensus.cc:491] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.049741 2720 raft_consensus.cc:513] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.050419 2720 leader_election.cc:290] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:49.967373 1245 catalog_manager.cc:5582] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:50.058527 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
I20250623 08:04:50.059145 1245 catalog_manager.cc:5582] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:50.059722 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:50.061062 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:50.061968 342 tablet_bootstrap.cc:492] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:50.062053 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:50.062487 342 ts_tablet_manager.cc:1397] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.216s user 0.008s sys 0.004s
I20250623 08:04:50.065960 2755 raft_consensus.cc:491] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.066110 2755 raft_consensus.cc:513] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.066684 2755 leader_election.cc:290] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.067662 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b8cfcbce6a472ab2daae28226cf766" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.068220 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f080a682e532406a83712b6f3e6d2f7b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.068377 32374 raft_consensus.cc:2466] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.074864 952 ts_tablet_manager.cc:1939] T db81250d1e164a7d94affd764fc0ff0c P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:50.077360 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet db81250d1e164a7d94affd764fc0ff0c (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]) successfully deleted
I20250623 08:04:50.079075 343 raft_consensus.cc:357] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.079259 343 raft_consensus.cc:383] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.079311 343 raft_consensus.cc:738] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.079489 343 consensus_queue.cc:260] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.079974 343 ts_tablet_manager.cc:1428] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.082s user 0.001s sys 0.001s
I20250623 08:04:50.080241 343 tablet_bootstrap.cc:492] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:50.083659 343 tablet_bootstrap.cc:654] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.106698 343 tablet_bootstrap.cc:492] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:50.106892 343 ts_tablet_manager.cc:1397] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.005s
I20250623 08:04:50.106886 1235 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 2b405f87bb1e4d0686a1f627835d901f
I20250623 08:04:50.107411 1235 catalog_manager.cc:6652] ReplaceTablet: tablet 2b405f87bb1e4d0686a1f627835d901f deleted and replaced by tablet 4294979f2ac848b5846d1aa6c879ccad
I20250623 08:04:50.107555 1235 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 55) took 2493 ms. Trace:
I20250623 08:04:50.107875 343 raft_consensus.cc:357] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.108083 343 raft_consensus.cc:383] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.107656 1235 rpcz_store.cc:276] 0623 08:04:47.613864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.969651 (+2355787us) service_pool.cc:225] Handling call
0623 08:04:50.107522 (+137871us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.970911 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.971037 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.971055 (+ 18us) write_op.cc:435] Acquired schema lock
0623 08:04:49.971059 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:49.971176 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.971189 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.971196 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.971283 (+ 87us) tablet.cc:699] Row locks acquired
0623 08:04:49.971288 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:49.971322 (+ 34us) write_op.cc:270] Start()
0623 08:04:49.971354 (+ 32us) write_op.cc:276] Timestamp: P: 1750665889971295 usec, L: 0
0623 08:04:49.971359 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.971542 (+ 183us) log.cc:844] Serialized 656 byte log entry
0623 08:04:50.104508 (+132966us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.104828 (+ 320us) write_op.cc:301] APPLY: starting
0623 08:04:50.104847 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:50.104896 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:50.104924 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:50.105024 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:50.105084 (+ 60us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:50.105092 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:50.105175 (+ 83us) log.cc:844] Serialized 29 byte log entry
0623 08:04:50.105245 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.105264 (+ 19us) write_op.cc:454] Released schema lock
0623 08:04:50.105273 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:50.105357 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.105363 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":265,"apply.run_cpu_time_us":517,"apply.run_wall_time_us":517,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":703,"replication_time_us":133112}]]}
I20250623 08:04:50.108141 343 raft_consensus.cc:738] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.108358 343 consensus_queue.cc:260] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.108803 1242 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 1c529e826fa84a919c018887620e4023
I20250623 08:04:50.108913 343 ts_tablet_manager.cc:1428] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:50.109229 343 tablet_bootstrap.cc:492] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:50.109244 1242 catalog_manager.cc:6652] ReplaceTablet: tablet 1c529e826fa84a919c018887620e4023 deleted and replaced by tablet 79104f3256dd42ce8e92adf090b5cc8e
I20250623 08:04:50.109352 1242 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 54) took 2569 ms. Trace:
I20250623 08:04:50.109443 1242 rpcz_store.cc:276] 0623 08:04:47.539484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.948827 (+2409343us) service_pool.cc:225] Handling call
0623 08:04:50.109319 (+160492us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.979645 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.979743 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.979748 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:49.979750 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:49.979834 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.979841 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.979845 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:49.979928 (+ 83us) tablet.cc:699] Row locks acquired
0623 08:04:49.979930 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:49.979956 (+ 26us) write_op.cc:270] Start()
0623 08:04:49.979987 (+ 31us) write_op.cc:276] Timestamp: P: 1750665889979931 usec, L: 0
0623 08:04:49.979992 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.980189 (+ 197us) log.cc:844] Serialized 622 byte log entry
0623 08:04:50.104624 (+124435us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.105878 (+ 1254us) write_op.cc:301] APPLY: starting
0623 08:04:50.105889 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:50.105932 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:50.105953 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:50.106057 (+ 104us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:50.106107 (+ 50us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:50.106113 (+ 6us) write_op.cc:312] APPLY: finished
0623 08:04:50.106164 (+ 51us) log.cc:844] Serialized 29 byte log entry
0623 08:04:50.106204 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.106219 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:50.106225 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1224,"apply.run_cpu_time_us":406,"apply.run_wall_time_us":406,"num_ops":2,"prepare.queue_time_us":1159,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":973,"replication_time_us":124615,"threads_started":1}]]}
I20250623 08:04:50.111202 343 tablet_bootstrap.cc:654] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.111793 32350 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b405f87bb1e4d0686a1f627835d901f with delete_type TABLET_DATA_DELETED (replaced by tablet 4294979f2ac848b5846d1aa6c879ccad) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:50.112023 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c529e826fa84a919c018887620e4023 with delete_type TABLET_DATA_DELETED (replaced by tablet 79104f3256dd42ce8e92adf090b5cc8e) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:50.112258 32481 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b405f87bb1e4d0686a1f627835d901f with delete_type TABLET_DATA_DELETED (replaced by tablet 4294979f2ac848b5846d1aa6c879ccad) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:50.112501 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c529e826fa84a919c018887620e4023 with delete_type TABLET_DATA_DELETED (replaced by tablet 79104f3256dd42ce8e92adf090b5cc8e) from {username='slave'} at 127.0.0.1:34076
W20250623 08:04:50.112586 32589 leader_election.cc:343] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.112793 2749 tablet_replica.cc:331] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:50.112798 32589 leader_election.cc:304] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.112996 2558 raft_consensus.cc:2747] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.113013 2749 raft_consensus.cc:2241] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:50.113190 2749 raft_consensus.cc:2270] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:50.113736 2749 ts_tablet_manager.cc:1905] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:50.120474 343 tablet_bootstrap.cc:492] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:50.120609 343 ts_tablet_manager.cc:1397] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250623 08:04:50.121243 343 raft_consensus.cc:357] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.121398 343 raft_consensus.cc:383] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.121447 343 raft_consensus.cc:738] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.121618 343 consensus_queue.cc:260] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.111268 2718 raft_consensus.cc:491] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.121986 2718 raft_consensus.cc:513] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.122802 2718 leader_election.cc:290] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.123265 2720 raft_consensus.cc:491] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.123446 2720 raft_consensus.cc:513] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.123864 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b405f87bb1e4d0686a1f627835d901f with delete_type TABLET_DATA_DELETED (replaced by tablet 4294979f2ac848b5846d1aa6c879ccad) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:50.123963 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c529e826fa84a919c018887620e4023 with delete_type TABLET_DATA_DELETED (replaced by tablet 79104f3256dd42ce8e92adf090b5cc8e) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:50.124114 2720 leader_election.cc:290] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.122156 343 ts_tablet_manager.cc:1428] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250623 08:04:50.124768 343 tablet_bootstrap.cc:492] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:50.125073 2749 ts_tablet_manager.cc:1918] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:50.125473 2749 log.cc:1199] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/2b405f87bb1e4d0686a1f627835d901f
I20250623 08:04:50.125967 2749 ts_tablet_manager.cc:1939] T 2b405f87bb1e4d0686a1f627835d901f P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:50.126318 343 tablet_bootstrap.cc:654] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.129124 2659 ts_tablet_manager.cc:1918] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250623 08:04:50.132822 32368 raft_consensus.cc:3058] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.135423 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:50.139082 342 raft_consensus.cc:357] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.139394 342 raft_consensus.cc:383] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.139505 342 raft_consensus.cc:738] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.139815 342 consensus_queue.cc:260] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.140600 342 ts_tablet_manager.cc:1428] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.078s user 0.002s sys 0.000s
I20250623 08:04:50.140982 342 tablet_bootstrap.cc:492] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:50.142846 342 tablet_bootstrap.cc:654] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.146160 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.146716 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.136029 1235 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_78e064bbf114486f9cfa40945fb67bf2" } modify_external_catalogs: true
I20250623 08:04:50.147898 1235 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_78e064bbf114486f9cfa40945fb67bf2" } modify_external_catalogs: true
I20250623 08:04:50.136382 1242 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57678:
table { table_name: "default.Table_25f75684255a4b99a8aff38ee935aac5" } new_table_name: "default.Table_9eed3165e01347d3a7492a52966d849f" modify_external_catalogs: true
I20250623 08:04:50.146543 1213 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 2b405f87bb1e4d0686a1f627835d901f (table default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]) successfully deleted
W20250623 08:04:50.149901 1213 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 1c529e826fa84a919c018887620e4023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c529e826fa84a919c018887620e4023 already in progress: creating tablet
I20250623 08:04:50.151286 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 11bb55088ab24b94af3e509810890247 (table default.Table_e4c31249050c4016891036cf9507b607 [id=9a7a76e61e1b43508b6f6080f5163417]). Expected version 1 got 0
I20250623 08:04:50.152807 1245 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:50.154245 2659 log.cc:1199] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/886fc0fa39c54bddb25ac01f68d95805
I20250623 08:04:50.158887 2659 ts_tablet_manager.cc:1939] T 886fc0fa39c54bddb25ac01f68d95805 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:50.165831 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 886fc0fa39c54bddb25ac01f68d95805 (table default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]) successfully deleted
I20250623 08:04:50.166800 2790 raft_consensus.cc:491] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.166949 2790 raft_consensus.cc:513] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.167591 2790 leader_election.cc:290] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.168658 2659 tablet_replica.cc:331] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:50.177412 2659 raft_consensus.cc:2241] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
W20250623 08:04:50.180917 32589 leader_election.cc:343] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:50.181682 32589 leader_election.cc:343] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:50.182264 32589 leader_election.cc:343] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.183248 2659 raft_consensus.cc:2270] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:50.194689 32376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30dfa986dfd44658a0ffb07796030520" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.194936 32376 raft_consensus.cc:2466] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.195734 32345 tablet_service.cc:1468] Processing CreateTablet for tablet c75c8396c86d495496839adebbbd476c (DEFAULT_TABLE table=default.Table_dbe34c2ebf43478fb748f8e4d89c083e [id=c3c06f7715714ad9a79ecca1e08f52de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:50.196106 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75c8396c86d495496839adebbbd476c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250623 08:04:50.228437 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 44704.1us
I20250623 08:04:50.228543 32630 tablet_service.cc:1468] Processing CreateTablet for tablet c75c8396c86d495496839adebbbd476c (DEFAULT_TABLE table=default.Table_dbe34c2ebf43478fb748f8e4d89c083e [id=c3c06f7715714ad9a79ecca1e08f52de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:50.229012 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75c8396c86d495496839adebbbd476c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:50.228614 2718 raft_consensus.cc:491] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.229225 2718 raft_consensus.cc:513] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.229947 2718 leader_election.cc:290] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.230898 2659 ts_tablet_manager.cc:1905] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250623 08:04:50.247711 32589 leader_election.cc:343] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.250830 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41b15c89a744a698ca5eee2f143b4bc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.251480 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4836d621112e46c380c7bc2b7a93117f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.262550 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "492ffaeaaa314dcfb7512d8f23946a67" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.263221 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88f2a63b5a734c8f9b7a53361590245d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:50.263438 32377 raft_consensus.cc:3058] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.264333 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3ad3d261a794d1184061c8e9b384693" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.265039 339 tablet_bootstrap.cc:492] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.265197 339 ts_tablet_manager.cc:1397] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.281s user 0.055s sys 0.053s
I20250623 08:04:50.267310 32377 raft_consensus.cc:2466] T 88f2a63b5a734c8f9b7a53361590245d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.143239 2720 raft_consensus.cc:491] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.269827 2720 raft_consensus.cc:513] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.270385 2720 leader_election.cc:290] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.150241 952 tablet_replica.cc:331] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:50.195366 32482 tablet_service.cc:1468] Processing CreateTablet for tablet c75c8396c86d495496839adebbbd476c (DEFAULT_TABLE table=default.Table_dbe34c2ebf43478fb748f8e4d89c083e [id=c3c06f7715714ad9a79ecca1e08f52de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:50.271227 32482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c75c8396c86d495496839adebbbd476c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:50.273669 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1494f83d13cd49c7a804593f45f97cc6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.274118 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcfe68d4e6ae470d816ac4c494dcaa51" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.274277 32375 raft_consensus.cc:2466] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.219588 1359 raft_consensus.cc:491] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.274853 1359 raft_consensus.cc:513] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.275560 1359 leader_election.cc:290] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.238299 2439 raft_consensus.cc:491] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.275781 2439 raft_consensus.cc:513] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.276364 2439 leader_election.cc:290] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.276571 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eebc32fdfa44abcbd6f04a636d02c7c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.277045 952 raft_consensus.cc:2241] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:50.277432 952 raft_consensus.cc:2270] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:50.278467 339 raft_consensus.cc:357] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.278661 339 raft_consensus.cc:383] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.278738 339 raft_consensus.cc:738] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.278973 339 consensus_queue.cc:260] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.279526 339 ts_tablet_manager.cc:1428] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20250623 08:04:50.279712 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a07188a4d340bc9caf010d5202e34d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.279824 339 tablet_bootstrap.cc:492] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.281352 339 tablet_bootstrap.cc:654] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.285962 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e4cde2af694f07a9176313b4b93a46" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:50.286193 32375 raft_consensus.cc:3058] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.281867 952 ts_tablet_manager.cc:1905] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:50.289037 32375 raft_consensus.cc:2466] T c8e4cde2af694f07a9176313b4b93a46 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.307116 2790 raft_consensus.cc:491] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.307281 2790 raft_consensus.cc:513] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.308130 2790 leader_election.cc:290] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.308713 343 tablet_bootstrap.cc:492] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:50.308871 343 ts_tablet_manager.cc:1397] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.184s user 0.008s sys 0.022s
I20250623 08:04:50.309774 343 raft_consensus.cc:357] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.309974 343 raft_consensus.cc:383] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.310041 343 raft_consensus.cc:738] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.310297 343 consensus_queue.cc:260] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:50.315392 32589 leader_election.cc:343] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:50.316327 32589 leader_election.cc:343] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:50.316658 32589 leader_election.cc:343] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.316139 2790 raft_consensus.cc:491] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.317108 2790 raft_consensus.cc:513] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.317835 2790 leader_election.cc:290] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.317785 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b7416ad6a244d978457e529cbc789de" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.319638 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e27da2a90c1b43deb3dc90834f4e312e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:50.320091 32589 leader_election.cc:343] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.330060 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e3920ac85e54379abbcc42dd80db504" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.330292 32515 raft_consensus.cc:2391] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
W20250623 08:04:50.329855 32589 leader_election.cc:343] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.333341 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ae0a411a3e54c45b8d5c5b9b0eea762" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.334933 343 ts_tablet_manager.cc:1428] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20250623 08:04:50.335217 343 tablet_bootstrap.cc:492] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:50.337294 343 tablet_bootstrap.cc:654] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.340777 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.340968 32512 raft_consensus.cc:2466] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:50.341176 32519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bfad68d1f20486997ea1b0a8d97206b" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.341912 32515 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.342131 32515 raft_consensus.cc:2466] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:50.344103 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cee3de5953244f98428747bea912f77" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:50.353531 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 56381us
I20250623 08:04:50.354082 342 tablet_bootstrap.cc:492] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:50.354214 339 tablet_bootstrap.cc:492] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.354231 342 ts_tablet_manager.cc:1397] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.213s user 0.003s sys 0.000s
W20250623 08:04:50.354554 32314 leader_election.cc:343] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 92005abc61dd40ef9da8124cadc29b6a
I20250623 08:04:50.354995 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3ad3d261a794d1184061c8e9b384693" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.355073 342 raft_consensus.cc:357] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.355347 342 raft_consensus.cc:383] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.355432 2836 tablet.cc:1722] T 11bb55088ab24b94af3e509810890247 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.357352 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b213c8c685c43868df1e814c79fc61f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.354573 339 ts_tablet_manager.cc:1397] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.002s
I20250623 08:04:50.358881 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b213c8c685c43868df1e814c79fc61f" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.359107 32642 raft_consensus.cc:2391] T 1b213c8c685c43868df1e814c79fc61f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:50.359619 952 ts_tablet_manager.cc:1918] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:50.359923 952 log.cc:1199] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/d19d468ce13d4108a9badc36fc503306
I20250623 08:04:50.360780 952 ts_tablet_manager.cc:1939] T d19d468ce13d4108a9badc36fc503306 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:50.361933 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet d19d468ce13d4108a9badc36fc503306 (table default.Table_5c43b8626a6640e4a61e495ab2ea5697 [id=46664e505d9648ca889484123253a613]) successfully deleted
I20250623 08:04:50.362516 952 tablet_replica.cc:331] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:50.362896 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41b15c89a744a698ca5eee2f143b4bc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.355563 342 raft_consensus.cc:738] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.363801 339 raft_consensus.cc:357] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.363787 342 consensus_queue.cc:260] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.364923 342 ts_tablet_manager.cc:1428] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250623 08:04:50.365551 342 tablet_bootstrap.cc:492] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:50.366817 32512 raft_consensus.cc:2466] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:50.368230 339 raft_consensus.cc:383] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.368458 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4836d621112e46c380c7bc2b7a93117f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.368770 339 raft_consensus.cc:738] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.369037 339 consensus_queue.cc:260] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.369383 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b7416ad6a244d978457e529cbc789de" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.370159 32314 leader_election.cc:304] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:50.370924 2446 raft_consensus.cc:2802] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 2 FOLLOWER]: Leader election won for term 2
I20250623 08:04:50.371026 2446 raft_consensus.cc:695] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [term 2 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:50.371230 2446 consensus_queue.cc:237] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.374398 32427 heartbeater.cc:344] Connected to a master server at 127.31.69.62:39465
I20250623 08:04:50.374589 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:50.376837 32314 leader_election.cc:304] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:50.370716 339 ts_tablet_manager.cc:1428] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250623 08:04:50.377240 339 tablet_bootstrap.cc:492] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.377486 2659 ts_tablet_manager.cc:1918] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:50.377614 2659 log.cc:1199] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/9bfa787f828b411ab68d81bb896f7fb6
I20250623 08:04:50.378003 2659 ts_tablet_manager.cc:1939] T 9bfa787f828b411ab68d81bb896f7fb6 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:50.378139 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e27da2a90c1b43deb3dc90834f4e312e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.378286 32510 raft_consensus.cc:2466] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:50.378924 339 tablet_bootstrap.cc:654] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.384590 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee82db25aa544a21a1419e081196f179" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.384840 32648 raft_consensus.cc:2466] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.385790 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36d21fe528f5410eacd655bf15306b98" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.386047 32641 raft_consensus.cc:2466] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.390833 2788 raft_consensus.cc:2747] T 1b213c8c685c43868df1e814c79fc61f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.392630 952 raft_consensus.cc:2241] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:50.393257 952 raft_consensus.cc:2270] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:50.394191 2755 raft_consensus.cc:491] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.394316 2755 raft_consensus.cc:513] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.394531 952 ts_tablet_manager.cc:1905] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:50.394860 2755 leader_election.cc:290] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.397475 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.397637 32653 raft_consensus.cc:2466] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.397763 2659 tablet_replica.cc:331] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:50.397768 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 9bfa787f828b411ab68d81bb896f7fb6 (table default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]) successfully deleted
I20250623 08:04:50.398167 2659 raft_consensus.cc:2241] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:50.399178 2659 raft_consensus.cc:2270] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:50.399262 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee82db25aa544a21a1419e081196f179" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.399914 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36d21fe528f5410eacd655bf15306b98" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:50.400014 32589 leader_election.cc:343] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.400125 32589 leader_election.cc:304] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.400380 2659 ts_tablet_manager.cc:1905] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:50.400454 2517 raft_consensus.cc:2747] T 1494f83d13cd49c7a804593f45f97cc6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.404126 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.404773 2517 raft_consensus.cc:491] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.404891 2517 raft_consensus.cc:513] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.405449 2517 leader_election.cc:290] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.406236 342 tablet_bootstrap.cc:654] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:50.406554 32589 leader_election.cc:343] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 2c5ecd4fa5574a1b90b6e82f89ee88f7
I20250623 08:04:50.407588 343 tablet_bootstrap.cc:492] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:50.407835 343 ts_tablet_manager.cc:1397] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.003s
I20250623 08:04:50.410884 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.411046 32372 raft_consensus.cc:2466] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.410878 343 raft_consensus.cc:357] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.411633 343 raft_consensus.cc:383] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.412019 343 raft_consensus.cc:738] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.413671 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.413868 32653 raft_consensus.cc:2466] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.412952 343 consensus_queue.cc:260] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.415174 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e27da2a90c1b43deb3dc90834f4e312e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.415339 32653 raft_consensus.cc:2466] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:50.411685 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecc73f52f0bc43c5bbdca2b89deecb21" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.417121 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecc73f52f0bc43c5bbdca2b89deecb21" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.418443 32314 leader_election.cc:304] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:50.419354 2446 raft_consensus.cc:2747] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.419623 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e27da2a90c1b43deb3dc90834f4e312e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.419463 343 ts_tablet_manager.cc:1428] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20250623 08:04:50.420994 339 tablet_bootstrap.cc:492] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.421196 339 ts_tablet_manager.cc:1397] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.002s
I20250623 08:04:50.421726 2517 raft_consensus.cc:491] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.421860 2517 raft_consensus.cc:513] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.422469 2517 leader_election.cc:290] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.423395 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9a0c57133884896b03e9c0a8afb8bde" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.423715 339 raft_consensus.cc:357] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.423933 339 raft_consensus.cc:383] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250623 08:04:50.423918 32589 leader_election.cc:343] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.424047 339 raft_consensus.cc:738] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.424278 339 consensus_queue.cc:260] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.424804 339 ts_tablet_manager.cc:1428] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:50.424799 32589 leader_election.cc:304] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.425077 2517 raft_consensus.cc:2747] T 4e3920ac85e54379abbcc42dd80db504 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.425258 339 tablet_bootstrap.cc:492] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
W20250623 08:04:50.425484 32589 consensus_peers.cc:487] T db81250d1e164a7d94affd764fc0ff0c P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db81250d1e164a7d94affd764fc0ff0c. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:50.425792 2517 raft_consensus.cc:491] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7351aa31800245659edc9837e27daffd)
I20250623 08:04:50.425966 2517 raft_consensus.cc:513] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.426512 2517 leader_election.cc:290] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.426915 339 tablet_bootstrap.cc:654] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:50.427376 32589 leader_election.cc:343] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: f83f68ebda7e4826bbb17278cd4ac5a8
I20250623 08:04:50.432384 343 tablet_bootstrap.cc:492] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
W20250623 08:04:50.435665 32589 leader_election.cc:343] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.435802 32589 leader_election.cc:304] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.436061 2558 raft_consensus.cc:2747] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.439690 339 tablet_bootstrap.cc:492] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.439870 339 ts_tablet_manager.cc:1397] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20250623 08:04:50.440560 339 raft_consensus.cc:357] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.440785 339 raft_consensus.cc:383] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.440873 339 raft_consensus.cc:738] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.441130 339 consensus_queue.cc:260] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.441745 339 ts_tablet_manager.cc:1428] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:50.442117 339 tablet_bootstrap.cc:492] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.445916 32589 leader_election.cc:304] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.446283 2558 raft_consensus.cc:2802] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.446385 2558 raft_consensus.cc:491] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.446466 2558 raft_consensus.cc:3058] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:50.448627 32589 leader_election.cc:343] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.448869 32589 leader_election.cc:304] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.449209 2718 raft_consensus.cc:2747] T 3bfad68d1f20486997ea1b0a8d97206b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.449293 2558 raft_consensus.cc:513] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.450008 2558 leader_election.cc:290] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.450527 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
W20250623 08:04:50.451212 32589 leader_election.cc:343] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.452697 339 tablet_bootstrap.cc:654] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.452610 32589 leader_election.cc:304] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.452971 2558 raft_consensus.cc:2802] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.453112 2558 raft_consensus.cc:491] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.453256 2558 raft_consensus.cc:3058] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.453195 343 tablet_bootstrap.cc:654] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:50.453821 32589 leader_election.cc:343] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.454542 32589 leader_election.cc:304] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.456049 2558 raft_consensus.cc:513] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.456712 2718 raft_consensus.cc:2747] T 1cee3de5953244f98428747bea912f77 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.456996 2558 leader_election.cc:290] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.459079 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:50.459302 32368 raft_consensus.cc:3058] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.462113 32368 raft_consensus.cc:2466] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:50.462618 342 tablet_bootstrap.cc:492] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:50.462857 342 ts_tablet_manager.cc:1397] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.000s
I20250623 08:04:50.463891 342 raft_consensus.cc:357] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.464180 342 raft_consensus.cc:383] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.464298 342 raft_consensus.cc:738] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.464596 342 consensus_queue.cc:260] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.465330 342 ts_tablet_manager.cc:1428] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:50.465708 342 tablet_bootstrap.cc:492] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:50.466897 339 tablet_bootstrap.cc:492] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.467052 339 ts_tablet_manager.cc:1397] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
I20250623 08:04:50.467722 339 raft_consensus.cc:357] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.468009 339 raft_consensus.cc:383] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.468112 339 raft_consensus.cc:738] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.468391 339 consensus_queue.cc:260] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:50.468381 32589 leader_election.cc:343] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.469091 339 ts_tablet_manager.cc:1428] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:50.469285 342 tablet_bootstrap.cc:654] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.469403 339 tablet_bootstrap.cc:492] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.469177 32589 leader_election.cc:304] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.470367 2774 raft_consensus.cc:2747] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.470901 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
W20250623 08:04:50.477635 32453 leader_election.cc:343] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.471262 339 tablet_bootstrap.cc:654] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.478344 32589 leader_election.cc:304] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.478624 32453 leader_election.cc:304] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:50.479513 2820 raft_consensus.cc:2802] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.479594 2820 raft_consensus.cc:491] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.479660 2820 raft_consensus.cc:3058] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.479629 2774 raft_consensus.cc:2802] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.479746 2774 raft_consensus.cc:491] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.479888 2774 raft_consensus.cc:3058] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:50.480754 32589 leader_election.cc:343] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.481271 1823 consensus_queue.cc:1035] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:50.481801 1823 consensus_queue.cc:1035] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:50.482141 32589 leader_election.cc:304] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.482868 2820 raft_consensus.cc:513] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:50.482805 32453 leader_election.cc:343] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.483600 2820 leader_election.cc:290] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.483618 2718 raft_consensus.cc:2747] T 4836d621112e46c380c7bc2b7a93117f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.484051 2774 raft_consensus.cc:513] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.484587 2774 leader_election.cc:290] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.488648 339 tablet_bootstrap.cc:492] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.488826 339 ts_tablet_manager.cc:1397] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.003s
I20250623 08:04:50.489537 339 raft_consensus.cc:357] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.489768 339 raft_consensus.cc:383] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.489884 339 raft_consensus.cc:738] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.490097 339 consensus_queue.cc:260] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.490650 339 ts_tablet_manager.cc:1428] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:50.492525 32453 leader_election.cc:304] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:50.492718 32589 leader_election.cc:343] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.493626 2439 raft_consensus.cc:2802] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.493701 2439 raft_consensus.cc:491] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.493783 2439 raft_consensus.cc:3058] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.496410 2439 raft_consensus.cc:513] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.496703 339 tablet_bootstrap.cc:492] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.496927 2439 leader_election.cc:290] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.493461 32589 leader_election.cc:304] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.497882 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:50.498308 2774 raft_consensus.cc:2747] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.498546 339 tablet_bootstrap.cc:654] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.499131 2836 tablet.cc:1722] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.500398 32589 leader_election.cc:304] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.501929 2774 raft_consensus.cc:2802] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.502025 2774 raft_consensus.cc:491] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.502103 2774 raft_consensus.cc:3058] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.503157 2836 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 69) took 1012 ms. Trace:
I20250623 08:04:50.503363 2820 consensus_queue.cc:1035] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:50.503275 2836 rpcz_store.cc:276] 0623 08:04:49.490190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490300 (+ 110us) service_pool.cc:225] Handling call
0623 08:04:50.503141 (+1012841us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.575386 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.575492 (+ 106us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.575501 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.575508 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.575540 (+ 32us) alter_schema_op.cc:128] START. Timestamp: 1750665889575466
0623 08:04:49.575545 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.575697 (+ 152us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.499004 (+923307us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.499110 (+ 106us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.499503 (+ 393us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.502449 (+ 2946us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.502732 (+ 283us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.503099 (+ 367us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.503104 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":35,"prepare.queue_time_us":84835,"prepare.run_cpu_time_us":2260,"prepare.run_wall_time_us":19359,"raft.queue_time_us":229,"raft.run_cpu_time_us":498,"raft.run_wall_time_us":497,"replication_time_us":923416,"thread_start_us":95794,"threads_started":2,"wal-append.queue_time_us":10153}]]}
I20250623 08:04:50.504426 2836 tablet.cc:1722] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.505018 2774 raft_consensus.cc:513] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.505548 2774 leader_election.cc:290] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.505846 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:50.506911 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:50.509550 952 ts_tablet_manager.cc:1918] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:50.510870 952 log.cc:1199] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a383d10d53b04d5699dd3c4bad8878c7
I20250623 08:04:50.511860 2836 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 60) took 1022 ms. Trace:
I20250623 08:04:50.512851 2836 rpcz_store.cc:276] 0623 08:04:49.489317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.489656 (+ 339us) service_pool.cc:225] Handling call
0623 08:04:50.511844 (+1022188us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.572514 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.572636 (+ 122us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.572647 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.572655 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.572700 (+ 45us) alter_schema_op.cc:128] START. Timestamp: 1750665889572624
0623 08:04:49.572705 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.572889 (+ 184us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.499477 (+926588us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.504410 (+ 4933us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.504745 (+ 335us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.511386 (+ 6641us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.511617 (+ 231us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.511806 (+ 189us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.511811 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4892,"mutex_wait_us":8265,"prepare.queue_time_us":82489,"prepare.run_cpu_time_us":2137,"prepare.run_wall_time_us":21510,"raft.queue_time_us":8357,"raft.run_cpu_time_us":569,"raft.run_wall_time_us":569,"replication_time_us":926739,"spinlock_wait_cycles":21248,"thread_start_us":78273,"threads_started":3,"wal-append.queue_time_us":12245}]]}
I20250623 08:04:50.513880 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41b15c89a744a698ca5eee2f143b4bc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:50.514410 2836 tablet.cc:1722] T 23928d64bcef4c06af81698e688b7f94 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.515143 2847 tablet.cc:1722] T fd7f3d2e67e84bfcbd45a21658f69e7b P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.517148 32589 leader_election.cc:304] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:50.517289 32453 leader_election.cc:343] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.519102 2558 raft_consensus.cc:2802] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:50.519527 2558 raft_consensus.cc:695] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:50.519935 2558 consensus_queue.cc:237] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:50.521070 32589 leader_election.cc:343] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.522145 32453 leader_election.cc:304] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:50.523109 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e27da2a90c1b43deb3dc90834f4e312e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:50.523872 342 tablet_bootstrap.cc:492] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:50.523979 2820 raft_consensus.cc:2802] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:50.524402 2820 raft_consensus.cc:695] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:50.524708 2820 consensus_queue.cc:237] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.526088 32453 leader_election.cc:304] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:50.527964 952 ts_tablet_manager.cc:1939] T a383d10d53b04d5699dd3c4bad8878c7 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:50.528441 1823 raft_consensus.cc:2802] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.528517 1823 raft_consensus.cc:491] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.528587 1823 raft_consensus.cc:3058] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.530550 342 ts_tablet_manager.cc:1397] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.001s
I20250623 08:04:50.531529 1823 raft_consensus.cc:513] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
W20250623 08:04:50.531335 32453 leader_election.cc:343] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.532238 1823 leader_election.cc:290] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.532680 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
W20250623 08:04:50.532671 32589 leader_election.cc:343] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.534785 2851 tablet.cc:1722] T 077bb090b3ea4d029463304d7a6feee5 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:50.535014 32453 leader_election.cc:343] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.536396 2659 ts_tablet_manager.cc:1918] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:50.536641 2659 log.cc:1199] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/89bb7ea31b4d4ce7a3c063853142fbb2
I20250623 08:04:50.536903 2847 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 45) took 1048 ms. Trace:
I20250623 08:04:50.537223 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:50.536477 32453 leader_election.cc:304] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:50.538836 2820 raft_consensus.cc:2802] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:50.539050 2820 raft_consensus.cc:695] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:50.537685 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:50.539316 2820 consensus_queue.cc:237] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.540530 2836 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 47) took 1052 ms. Trace:
I20250623 08:04:50.540627 2836 rpcz_store.cc:276] 0623 08:04:49.488154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.488230 (+ 76us) service_pool.cc:225] Handling call
0623 08:04:50.540515 (+1052285us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.492445 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.492560 (+ 115us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.492567 (+ 7us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.492575 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.492611 (+ 36us) alter_schema_op.cc:128] START. Timestamp: 1750665889492545
0623 08:04:49.492613 (+ 2us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.492797 (+ 184us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.512680 (+1019883us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.514392 (+ 1712us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.515344 (+ 952us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.518223 (+ 2879us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.518442 (+ 219us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.540474 (+ 22032us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.540479 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1669,"prepare.queue_time_us":3774,"prepare.run_cpu_time_us":10499,"prepare.run_wall_time_us":82893,"replication_time_us":1020031,"thread_start_us":82434,"threads_started":1,"wal-append.queue_time_us":81825}]]}
I20250623 08:04:50.542249 2836 tablet.cc:1722] T b0d1ec9d56ba40128325a50f204fe920 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.542914 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 48) took 1054 ms. Trace:
I20250623 08:04:50.543066 952 rpcz_store.cc:276] 0623 08:04:49.488244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.488720 (+ 476us) service_pool.cc:225] Handling call
0623 08:04:50.362386 (+873666us) ts_tablet_manager.cc:1145] Deleting tablet a383d10d53b04d5699dd3c4bad8878c7
0623 08:04:50.362397 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.443343 (+ 80946us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.508724 (+ 65381us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.542892 (+ 34168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":873435}
I20250623 08:04:50.546141 2659 ts_tablet_manager.cc:1939] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:50.546993 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:50.547400 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:50.554901 339 tablet_bootstrap.cc:492] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.555109 339 ts_tablet_manager.cc:1397] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.004s
I20250623 08:04:50.555935 339 raft_consensus.cc:357] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.556174 339 raft_consensus.cc:383] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.556270 339 raft_consensus.cc:738] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.556517 339 consensus_queue.cc:260] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.557155 339 ts_tablet_manager.cc:1428] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:50.557469 339 tablet_bootstrap.cc:492] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.559247 339 tablet_bootstrap.cc:654] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.560173 952 tablet_replica.cc:331] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:50.560250 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet a383d10d53b04d5699dd3c4bad8878c7 (table default.Table_68d8428833794f57b663cdae5b7d6cb3 [id=bcf8b85a65544390b35245e6ce6f7336]) successfully deleted
I20250623 08:04:50.561617 343 tablet_bootstrap.cc:492] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:50.561853 343 ts_tablet_manager.cc:1397] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.130s user 0.018s sys 0.002s
I20250623 08:04:50.563217 343 raft_consensus.cc:357] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.563551 343 raft_consensus.cc:383] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.563743 343 raft_consensus.cc:738] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.564170 343 consensus_queue.cc:260] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.565061 952 raft_consensus.cc:2241] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:50.565364 952 raft_consensus.cc:2270] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:50.566195 343 ts_tablet_manager.cc:1428] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250623 08:04:50.567837 343 tablet_bootstrap.cc:492] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:50.569754 2774 raft_consensus.cc:491] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.569907 2774 raft_consensus.cc:513] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.570546 2774 leader_election.cc:290] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.570865 952 ts_tablet_manager.cc:1905] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:50.570863 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9604d4890c940d8992eaf30993bf622" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:50.571491 32589 leader_election.cc:343] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.572517 2849 tablet.cc:1722] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:50.545595 32453 leader_election.cc:343] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.537002 2847 rpcz_store.cc:276] 0623 08:04:49.487970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.488785 (+ 815us) service_pool.cc:225] Handling call
0623 08:04:50.536886 (+1048101us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.555698 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.555844 (+ 146us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.555856 (+ 12us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.555867 (+ 11us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.555913 (+ 46us) alter_schema_op.cc:128] START. Timestamp: 1750665889555835
0623 08:04:49.555918 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.556153 (+ 235us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.513083 (+956930us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.515121 (+ 2038us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.518443 (+ 3322us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.533533 (+ 15090us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.533786 (+ 253us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.536848 (+ 3062us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.536853 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1972,"prepare.queue_time_us":66533,"prepare.run_cpu_time_us":2899,"prepare.run_wall_time_us":21905,"raft.queue_time_us":33,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":341,"replication_time_us":957130,"thread_start_us":256473,"threads_started":3,"wal-append.queue_time_us":21370}]]}
W20250623 08:04:50.573791 32453 leader_election.cc:343] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.574026 2847 tablet.cc:1722] T 37b38cb54d6c45308358f556a692eabe P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.574309 1957 consensus_queue.cc:1035] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:50.574410 1823 consensus_queue.cc:1035] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:50.577878 2849 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 71) took 1087 ms. Trace:
I20250623 08:04:50.577992 32428 heartbeater.cc:499] Master 127.31.69.61:33313 was elected leader, sending a full tablet report...
I20250623 08:04:50.549326 342 raft_consensus.cc:357] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.578255 342 raft_consensus.cc:383] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.578353 342 raft_consensus.cc:738] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.578426 2847 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 66) took 1088 ms. Trace:
I20250623 08:04:50.578591 342 consensus_queue.cc:260] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.579573 2659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42984 (request call id 47) took 1087 ms. Trace:
I20250623 08:04:50.581349 342 ts_tablet_manager.cc:1428] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.051s user 0.002s sys 0.000s
I20250623 08:04:50.581713 342 tablet_bootstrap.cc:492] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
W20250623 08:04:50.582865 32453 leader_election.cc:343] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.584676 32453 leader_election.cc:304] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:50.585108 1823 raft_consensus.cc:2802] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.585178 1823 raft_consensus.cc:491] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.585237 1823 raft_consensus.cc:3058] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:50.585798 32453 leader_election.cc:343] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.586699 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
I20250623 08:04:50.587857 339 tablet_bootstrap.cc:492] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.588040 339 ts_tablet_manager.cc:1397] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20250623 08:04:50.588519 343 tablet_bootstrap.cc:654] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.588864 339 raft_consensus.cc:357] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.589025 339 raft_consensus.cc:383] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.589114 339 raft_consensus.cc:738] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.589325 339 consensus_queue.cc:260] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.589737 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:50.589927 339 ts_tablet_manager.cc:1428] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:50.594274 2659 rpcz_store.cc:276] 0623 08:04:49.492538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.493005 (+ 467us) service_pool.cc:225] Handling call
0623 08:04:50.397636 (+904631us) ts_tablet_manager.cc:1145] Deleting tablet 89bb7ea31b4d4ce7a3c063853142fbb2
0623 08:04:50.397682 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.452435 (+ 54753us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.536267 (+ 83832us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.579530 (+ 43263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":904467}
I20250623 08:04:50.594718 339 tablet_bootstrap.cc:492] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.595403 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:50.596105 2659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42984 (request call id 51) took 1082 ms. Trace:
I20250623 08:04:50.596563 339 tablet_bootstrap.cc:654] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.596472 2659 rpcz_store.cc:276] 0623 08:04:49.513802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.513894 (+ 92us) service_pool.cc:225] Handling call
0623 08:04:50.595349 (+1081455us) ts_tablet_manager.cc:1145] Deleting tablet 7ef27bb5b1764ef09888fb0dfcae5be7
0623 08:04:50.595465 (+ 116us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.595966 (+ 501us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081247}
I20250623 08:04:50.597589 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 89bb7ea31b4d4ce7a3c063853142fbb2 (table default.Table_28d324870b1049289c32fbfcf8c44ebf [id=cf51a9276ee0490c920bebfef8fbfaf8]) successfully deleted
W20250623 08:04:50.597880 1213 catalog_manager.cc:4908] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): delete failed for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ef27bb5b1764ef09888fb0dfcae5be7 already in progress: creating tablet
I20250623 08:04:50.598059 2659 tablet_replica.cc:331] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:50.598109 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:50.598955 2659 raft_consensus.cc:2241] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:50.577999 2849 rpcz_store.cc:276] 0623 08:04:49.490367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490438 (+ 71us) service_pool.cc:225] Handling call
0623 08:04:50.577862 (+1087424us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.577623 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.577698 (+ 75us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.577705 (+ 7us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.577712 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.577746 (+ 34us) alter_schema_op.cc:128] START. Timestamp: 1750665889577677
0623 08:04:49.577749 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.577868 (+ 119us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.525088 (+947220us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.572493 (+ 47405us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.573119 (+ 626us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.577390 (+ 4271us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.577639 (+ 249us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.577824 (+ 185us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.577829 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47342,"prepare.queue_time_us":86893,"prepare.run_cpu_time_us":3850,"prepare.run_wall_time_us":22886,"raft.queue_time_us":400,"raft.run_cpu_time_us":485,"raft.run_wall_time_us":484,"replication_time_us":947312,"thread_start_us":256282,"threads_started":2,"wal-append.queue_time_us":22574}]]}
I20250623 08:04:50.601845 2849 tablet.cc:1722] T 16bc969d09f74b0986070a504f9f670e P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.601693 2659 raft_consensus.cc:2270] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:50.602162 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:50.603364 2836 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 65) took 1113 ms. Trace:
I20250623 08:04:50.603689 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:50.608006 2659 ts_tablet_manager.cc:1905] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:50.578646 2847 rpcz_store.cc:276] 0623 08:04:49.489937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490029 (+ 92us) service_pool.cc:225] Handling call
0623 08:04:50.578380 (+1088351us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.573269 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.573366 (+ 97us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.573375 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.573384 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.573416 (+ 32us) alter_schema_op.cc:128] START. Timestamp: 1750665889573344
0623 08:04:49.573421 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.573594 (+ 173us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.544475 (+970881us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.574010 (+ 29535us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.574409 (+ 399us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.577810 (+ 3401us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.578056 (+ 246us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.578301 (+ 245us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.578308 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":29486,"mutex_wait_us":22127,"prepare.queue_time_us":82961,"prepare.run_cpu_time_us":33296,"prepare.run_wall_time_us":169857,"raft.queue_time_us":22716,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":208,"replication_time_us":971026,"thread_start_us":389338,"threads_started":2,"wal-append.queue_time_us":147311}]]}
I20250623 08:04:50.603441 2836 rpcz_store.cc:276] 0623 08:04:49.489848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.491266 (+ 1418us) service_pool.cc:225] Handling call
0623 08:04:50.603348 (+1112082us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.594079 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.594190 (+ 111us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.594201 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.594211 (+ 10us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.594249 (+ 38us) alter_schema_op.cc:128] START. Timestamp: 1750665889594224
0623 08:04:49.594253 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.594431 (+ 178us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.521691 (+927260us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.542233 (+ 20542us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.542627 (+ 394us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.602989 (+ 60362us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.603263 (+ 274us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.603306 (+ 43us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.603310 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":20490,"mutex_wait_us":1646,"prepare.queue_time_us":102494,"prepare.run_cpu_time_us":2231,"prepare.run_wall_time_us":14557,"raft.queue_time_us":1720,"raft.run_cpu_time_us":570,"raft.run_wall_time_us":568,"replication_time_us":927409,"thread_start_us":109745,"threads_started":3,"wal-append.queue_time_us":11730}]]}
I20250623 08:04:50.612774 2851 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 53) took 1124 ms. Trace:
I20250623 08:04:50.615151 342 tablet_bootstrap.cc:654] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.617662 339 tablet_bootstrap.cc:492] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.617861 339 ts_tablet_manager.cc:1397] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.006s
I20250623 08:04:50.618664 339 raft_consensus.cc:357] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.618950 339 raft_consensus.cc:383] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.619048 339 raft_consensus.cc:738] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.619340 339 consensus_queue.cc:260] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.620276 339 ts_tablet_manager.cc:1428] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:50.620585 339 tablet_bootstrap.cc:492] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.612150 1823 raft_consensus.cc:513] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.625308 1823 leader_election.cc:290] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.626634 339 tablet_bootstrap.cc:654] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.630160 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:50.612834 2718 raft_consensus.cc:491] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.630962 2718 raft_consensus.cc:513] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.631629 2718 leader_election.cc:290] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.631682 2446 raft_consensus.cc:491] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.631815 2446 raft_consensus.cc:513] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.618425 343 tablet_bootstrap.cc:492] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:50.632390 2446 leader_election.cc:290] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.632926 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a07188a4d340bc9caf010d5202e34d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.633138 32642 raft_consensus.cc:2466] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:50.633620 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c529e826fa84a919c018887620e4023" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:50.634126 32589 leader_election.cc:343] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.634606 32314 leader_election.cc:304] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:50.635020 2446 raft_consensus.cc:2802] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.635097 2446 raft_consensus.cc:491] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.635154 2446 raft_consensus.cc:3058] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.635197 2849 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 68) took 1145 ms. Trace:
I20250623 08:04:50.637781 2446 raft_consensus.cc:513] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.638363 2446 leader_election.cc:290] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.638935 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a07188a4d340bc9caf010d5202e34d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.639149 32510 raft_consensus.cc:2466] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:50.613986 32453 leader_election.cc:304] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.640442 1823 raft_consensus.cc:2802] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.640522 1823 raft_consensus.cc:491] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.640585 1823 raft_consensus.cc:3058] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.641094 32453 leader_election.cc:304] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.641582 1957 raft_consensus.cc:2802] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.641654 1957 raft_consensus.cc:491] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.641716 1957 raft_consensus.cc:3058] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.643291 32453 leader_election.cc:304] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.643493 2439 raft_consensus.cc:2747] T 4e3920ac85e54379abbcc42dd80db504 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:50.647308 32453 leader_election.cc:304] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.648108 1359 raft_consensus.cc:2802] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.648044 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a07188a4d340bc9caf010d5202e34d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.648186 1359 raft_consensus.cc:491] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.635290 2849 rpcz_store.cc:276] 0623 08:04:49.490103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490146 (+ 43us) service_pool.cc:225] Handling call
0623 08:04:50.635179 (+1145033us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.573992 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.574083 (+ 91us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.574092 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.574099 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.574137 (+ 38us) alter_schema_op.cc:128] START. Timestamp: 1750665889574063
0623 08:04:49.574142 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.574291 (+ 149us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.544578 (+970287us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.601824 (+ 57246us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.602267 (+ 443us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.634860 (+ 32593us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.635096 (+ 236us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.635135 (+ 39us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.635140 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":57197,"mutex_wait_us":21564,"prepare.queue_time_us":83603,"prepare.run_cpu_time_us":43116,"prepare.run_wall_time_us":183967,"raft.queue_time_us":22232,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":195,"replication_time_us":970412,"thread_start_us":273318,"threads_started":3,"wal-append.queue_time_us":147327}]]}
I20250623 08:04:50.648239 1359 raft_consensus.cc:3058] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.652737 952 ts_tablet_manager.cc:1918] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:50.654433 952 log.cc:1199] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/a34da529ffde4595a23fc6596b30e3e9
I20250623 08:04:50.654973 1957 raft_consensus.cc:513] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.654995 952 ts_tablet_manager.cc:1939] T a34da529ffde4595a23fc6596b30e3e9 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:50.655651 1957 leader_election.cc:290] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.656000 32642 raft_consensus.cc:3058] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.661903 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a07188a4d340bc9caf010d5202e34d" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:50.662052 32510 raft_consensus.cc:3058] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.662340 2820 consensus_queue.cc:1035] T 1b213c8c685c43868df1e814c79fc61f P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:50.659186 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 51fce46407ac4e57a74ad696e85daac7 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:50.663245 32642 raft_consensus.cc:2466] T a2a07188a4d340bc9caf010d5202e34d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:50.663584 32453 leader_election.cc:304] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.663691 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:50.663884 32314 leader_election.cc:304] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:50.664079 2820 raft_consensus.cc:2802] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.664198 2820 raft_consensus.cc:491] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.664258 2446 raft_consensus.cc:2802] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:50.664333 2820 raft_consensus.cc:3058] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.664592 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:50.664976 2446 raft_consensus.cc:695] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:50.665302 2446 consensus_queue.cc:237] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.666211 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:50.666585 339 tablet_bootstrap.cc:492] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.666818 339 ts_tablet_manager.cc:1397] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.013s
I20250623 08:04:50.666911 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7f890ee0747b4df0b82bc3b396191124 (table default.Table_7ccb5edd6b824f308d349c51d926d963 [id=9f97f6deca194137a353ea221d0e3676]). Expected version 1 got 0
I20250623 08:04:50.667968 339 raft_consensus.cc:357] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.668138 339 raft_consensus.cc:383] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.668226 339 raft_consensus.cc:738] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.668445 339 consensus_queue.cc:260] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.668995 339 ts_tablet_manager.cc:1428] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:50.612896 2851 rpcz_store.cc:276] 0623 08:04:49.488700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.491717 (+ 3017us) service_pool.cc:225] Handling call
0623 08:04:50.612759 (+1121042us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.594818 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.594912 (+ 94us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.594921 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.594929 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.594962 (+ 33us) alter_schema_op.cc:128] START. Timestamp: 1750665889594939
0623 08:04:49.594967 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.595121 (+ 154us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.516791 (+921670us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.534737 (+ 17946us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.535301 (+ 564us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.612316 (+ 77015us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.612677 (+ 361us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.612718 (+ 41us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.612722 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":17891,"prepare.queue_time_us":102833,"prepare.run_cpu_time_us":2274,"prepare.run_wall_time_us":13293,"raft.queue_time_us":87,"raft.run_cpu_time_us":467,"raft.run_wall_time_us":466,"replication_time_us":921791,"thread_start_us":96204,"threads_started":2,"wal-append.queue_time_us":11829}]]}
I20250623 08:04:50.671404 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:50.674279 339 tablet_bootstrap.cc:492] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.677868 32510 raft_consensus.cc:2466] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:50.680617 1823 raft_consensus.cc:513] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.687403 1823 leader_election.cc:290] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.694543 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 55) took 1205 ms. Trace:
I20250623 08:04:50.694646 952 rpcz_store.cc:276] 0623 08:04:49.488881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.488967 (+ 86us) service_pool.cc:225] Handling call
0623 08:04:50.560098 (+1071131us) ts_tablet_manager.cc:1145] Deleting tablet a34da529ffde4595a23fc6596b30e3e9
0623 08:04:50.560108 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.632794 (+ 72686us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.652629 (+ 19835us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.694522 (+ 41893us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4570,"tablet-delete.queue_time_us":1070961}
I20250623 08:04:50.695047 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 57) took 1206 ms. Trace:
I20250623 08:04:50.695127 952 rpcz_store.cc:276] 0623 08:04:49.489052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.489164 (+ 112us) service_pool.cc:225] Handling call
0623 08:04:50.694950 (+1205786us) ts_tablet_manager.cc:1145] Deleting tablet a9ea74d20c5f4eefbcb8508ed1774026
0623 08:04:50.694962 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.695034 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205664}
I20250623 08:04:50.695389 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 58) took 1206 ms. Trace:
I20250623 08:04:50.695454 952 rpcz_store.cc:276] 0623 08:04:49.489143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.489309 (+ 166us) service_pool.cc:225] Handling call
0623 08:04:50.695320 (+1206011us) ts_tablet_manager.cc:1145] Deleting tablet 3740dd120745493688eeb4c1ac311dcb
0623 08:04:50.695327 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.695377 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205882}
I20250623 08:04:50.695689 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 54) took 1206 ms. Trace:
I20250623 08:04:50.695767 952 rpcz_store.cc:276] 0623 08:04:49.488796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.489334 (+ 538us) service_pool.cc:225] Handling call
0623 08:04:50.695627 (+1206293us) ts_tablet_manager.cc:1145] Deleting tablet 1be8f8557b9a4ae2917c37b8d01cedf6
0623 08:04:50.695635 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.695678 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206110}
I20250623 08:04:50.696059 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 61) took 1206 ms. Trace:
I20250623 08:04:50.696141 952 rpcz_store.cc:276] 0623 08:04:49.489427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.489484 (+ 57us) service_pool.cc:225] Handling call
0623 08:04:50.695985 (+1206501us) ts_tablet_manager.cc:1145] Deleting tablet adc3cf4355c24b7395fcb22af8520346
0623 08:04:50.695995 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.696046 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206355}
I20250623 08:04:50.696729 952 tablet_replica.cc:331] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:50.695287 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet a34da529ffde4595a23fc6596b30e3e9 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]) successfully deleted
I20250623 08:04:50.697119 952 raft_consensus.cc:2241] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:50.697399 952 raft_consensus.cc:2270] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:50.697458 1213 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet a9ea74d20c5f4eefbcb8508ed1774026 because the tablet was not found. No further retry: Not found: Tablet not found: a9ea74d20c5f4eefbcb8508ed1774026
W20250623 08:04:50.697690 1213 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 3740dd120745493688eeb4c1ac311dcb because the tablet was not found. No further retry: Not found: Tablet not found: 3740dd120745493688eeb4c1ac311dcb
W20250623 08:04:50.697888 1213 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 1be8f8557b9a4ae2917c37b8d01cedf6 because the tablet was not found. No further retry: Not found: Tablet not found: 1be8f8557b9a4ae2917c37b8d01cedf6
I20250623 08:04:50.697919 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1cee3de5953244f98428747bea912f77 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]). Expected version 1 got 0
W20250623 08:04:50.698110 1213 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet adc3cf4355c24b7395fcb22af8520346 because the tablet was not found. No further retry: Not found: Tablet not found: adc3cf4355c24b7395fcb22af8520346
I20250623 08:04:50.698637 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ca5c694856f44201a8aa4314a4f81681 (table default.Table_682a42dbf21d43d49ad9d85b958b121c [id=4f81957f51bd40c192ad7c716cbd893f]). Expected version 1 got 0
I20250623 08:04:50.698983 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 035bb06bc7dd42f681886def9d56eb73 (table default.Table_53879d6f821f4167bf02ffbe967f73f7 [id=b3f407a7a26e4bd5ae9a719a5c4577c0]). Expected version 1 got 0
I20250623 08:04:50.699317 342 tablet_bootstrap.cc:492] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:50.699440 342 ts_tablet_manager.cc:1397] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.118s user 0.012s sys 0.001s
I20250623 08:04:50.699399 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:50.700050 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:50.700259 342 raft_consensus.cc:357] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.700412 342 raft_consensus.cc:383] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.700464 342 raft_consensus.cc:738] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.700608 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:50.700647 342 consensus_queue.cc:260] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.701591 342 ts_tablet_manager.cc:1428] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:50.702096 342 tablet_bootstrap.cc:492] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:50.687537 1359 raft_consensus.cc:513] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.698652 32453 leader_election.cc:304] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.703577 1359 leader_election.cc:290] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.704056 1823 raft_consensus.cc:2762] T a2a07188a4d340bc9caf010d5202e34d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250623 08:04:50.705941 952 ts_tablet_manager.cc:1905] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:50.707484 2851 tablet.cc:1722] T ecc73f52f0bc43c5bbdca2b89deecb21 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.711723 2849 tablet.cc:1722] T 89a24019b797467e857b744d701fb1ed P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.712194 2851 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 79) took 1067 ms. Trace:
I20250623 08:04:50.712357 2851 rpcz_store.cc:276] 0623 08:04:49.645185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.647358 (+ 2173us) service_pool.cc:225] Handling call
0623 08:04:50.712153 (+1064795us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.647784 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.647898 (+ 114us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.647909 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.647919 (+ 10us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.647961 (+ 42us) alter_schema_op.cc:128] START. Timestamp: 1750665889647928
0623 08:04:49.647966 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.648162 (+ 196us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.707322 (+1059160us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.707462 (+ 140us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.708080 (+ 618us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.711073 (+ 2993us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.711833 (+ 760us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.712115 (+ 282us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.712120 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":85,"prepare.queue_time_us":60,"prepare.run_cpu_time_us":12661,"prepare.run_wall_time_us":27089,"raft.queue_time_us":215,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":1059323,"thread_start_us":26312,"threads_started":1,"wal-append.queue_time_us":25795}]]}
I20250623 08:04:50.705554 2820 raft_consensus.cc:513] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.715407 2849 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 78) took 1084 ms. Trace:
I20250623 08:04:50.715539 2849 rpcz_store.cc:276] 0623 08:04:49.630800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.630932 (+ 132us) service_pool.cc:225] Handling call
0623 08:04:50.715393 (+1084461us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.633346 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:49.633458 (+ 112us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:49.633469 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:49.633477 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:49.633509 (+ 32us) alter_schema_op.cc:128] START. Timestamp: 1750665889633480
0623 08:04:49.633513 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.633693 (+ 180us) log.cc:844] Serialized 173 byte log entry
0623 08:04:50.711599 (+1077906us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.711706 (+ 107us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:50.712033 (+ 327us) log.cc:844] Serialized 5 byte log entry
0623 08:04:50.714954 (+ 2921us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.715177 (+ 223us) log.cc:844] Serialized 14 byte log entry
0623 08:04:50.715358 (+ 181us) alter_schema_op.cc:73] Released schema lock
0623 08:04:50.715362 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31,"prepare.queue_time_us":459,"prepare.run_cpu_time_us":4724,"prepare.run_wall_time_us":95993,"replication_time_us":1078055,"thread_start_us":95521,"threads_started":1,"wal-append.queue_time_us":94675}]]}
I20250623 08:04:50.722379 339 tablet_bootstrap.cc:654] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.723178 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:50.727680 342 tablet_bootstrap.cc:654] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.730096 2820 leader_election.cc:290] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.726959 2659 ts_tablet_manager.cc:1918] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:50.732666 2659 log.cc:1199] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/f97f14cbaffb46d681d8b11c0213cae4
I20250623 08:04:50.733301 2659 ts_tablet_manager.cc:1939] T f97f14cbaffb46d681d8b11c0213cae4 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:50.734433 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet f217a8eb12014fcf89b9df4a2cddf5f6 (table default.Table_9426e757e60c4d3a8d540d2e94b241d1 [id=86224d949cab4304b8bcd6b429415163]). Expected version 1 got 0
I20250623 08:04:50.735659 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:50.735951 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:50.736672 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c50655c06cd94c67a49513c39728eaa9 (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:50.736069 32453 leader_election.cc:304] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.738291 32453 leader_election.cc:304] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.738497 1957 raft_consensus.cc:2802] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.738610 1957 raft_consensus.cc:491] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.738689 2820 consensus_queue.cc:1035] T bcf753ede17f429b82612ad12e13185c P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:50.738946 2720 raft_consensus.cc:2802] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.739037 2720 raft_consensus.cc:491] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.739114 2720 raft_consensus.cc:3058] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.741864 2720 raft_consensus.cc:513] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.742440 2720 leader_election.cc:290] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.738696 1957 raft_consensus.cc:3058] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.745380 1957 raft_consensus.cc:513] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.746088 1957 leader_election.cc:290] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.754731 32453 leader_election.cc:304] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.756328 1957 raft_consensus.cc:2802] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.756417 1957 raft_consensus.cc:491] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.756482 1957 raft_consensus.cc:3058] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.759089 1957 raft_consensus.cc:513] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.760236 1957 leader_election.cc:290] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.761868 342 tablet_bootstrap.cc:492] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:50.762048 342 ts_tablet_manager.cc:1397] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20250623 08:04:50.763100 342 raft_consensus.cc:357] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.763342 342 raft_consensus.cc:383] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.763437 342 raft_consensus.cc:738] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.763708 342 consensus_queue.cc:260] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.764354 342 ts_tablet_manager.cc:1428] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:50.764665 342 tablet_bootstrap.cc:492] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:50.765978 2659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42984 (request call id 54) took 1242 ms. Trace:
I20250623 08:04:50.766085 2659 rpcz_store.cc:276] 0623 08:04:49.523699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.523775 (+ 76us) service_pool.cc:225] Handling call
0623 08:04:50.597644 (+1073869us) ts_tablet_manager.cc:1145] Deleting tablet f97f14cbaffb46d681d8b11c0213cae4
0623 08:04:50.597770 (+ 126us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.690694 (+ 92924us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.726680 (+ 35986us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.765950 (+ 39270us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1073644}
I20250623 08:04:50.766623 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet f97f14cbaffb46d681d8b11c0213cae4 (table default.Table_dd7b43110df2433597689f77b73907a1 [id=27942e98cc084670bc96433efb754c98]) successfully deleted
I20250623 08:04:50.767207 2659 tablet_replica.cc:331] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:50.767624 2659 raft_consensus.cc:2241] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:50.768288 2659 raft_consensus.cc:2270] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:50.769331 2659 ts_tablet_manager.cc:1905] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:50.769369 2446 consensus_queue.cc:1035] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:50.769706 339 tablet_bootstrap.cc:492] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.769863 339 ts_tablet_manager.cc:1397] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.000s
I20250623 08:04:50.770192 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:50.770562 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_8f362905029746c19e0e7ce1433c594d [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 2 got 0
I20250623 08:04:50.770715 339 raft_consensus.cc:357] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.770948 339 raft_consensus.cc:383] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.771055 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:50.771241 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_d281efc20a4542faa2838570433fd540 [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 1 got 0
I20250623 08:04:50.771217 339 raft_consensus.cc:738] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.771450 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:50.771705 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:50.771906 339 consensus_queue.cc:260] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.773187 339 ts_tablet_manager.cc:1428] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:50.773535 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:50.768469 32453 leader_election.cc:304] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:50.774536 342 tablet_bootstrap.cc:654] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.774401 339 tablet_bootstrap.cc:492] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:50.775945 1957 raft_consensus.cc:2802] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.776029 1957 raft_consensus.cc:491] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.776094 1957 raft_consensus.cc:3058] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.776432 2558 raft_consensus.cc:491] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59f7c614254f42009306b13337f06576)
I20250623 08:04:50.776588 2558 raft_consensus.cc:513] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.777192 2558 leader_election.cc:290] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
W20250623 08:04:50.778136 32589 leader_election.cc:343] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Not found: Tablet not found: 803d3607d75443568b7ddc9f0c0bdc90
I20250623 08:04:50.783538 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:50.784221 2446 raft_consensus.cc:491] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.784369 2446 raft_consensus.cc:513] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.784991 2446 leader_election.cc:290] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.785542 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:50.785717 32642 raft_consensus.cc:2466] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:50.786180 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.786402 32514 raft_consensus.cc:2391] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:50.787526 32314 leader_election.cc:304] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:50.789760 2446 raft_consensus.cc:2802] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:50.789873 2446 raft_consensus.cc:491] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:50.789994 2446 raft_consensus.cc:3058] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.790635 2558 raft_consensus.cc:491] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.790820 2558 raft_consensus.cc:513] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.791458 2558 leader_election.cc:290] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.792088 1957 raft_consensus.cc:513] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.792622 1957 leader_election.cc:290] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.793422 2446 raft_consensus.cc:513] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.794003 2446 leader_election.cc:290] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.794653 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aface2a39d764a6cb6e4965867ab7b38" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:50.795156 32589 leader_election.cc:343] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.796525 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.796675 32641 raft_consensus.cc:3058] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.796774 32372 raft_consensus.cc:3058] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:50.797971 1218 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4eebc32fdfa44abcbd6f04a636d02c7c consensus state has no leader
I20250623 08:04:50.800962 32641 raft_consensus.cc:2466] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:50.801316 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f080a682e532406a83712b6f3e6d2f7b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:50.801115 339 tablet_bootstrap.cc:654] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.801512 32368 raft_consensus.cc:3058] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.801992 2788 consensus_queue.cc:1035] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:50.802047 32314 leader_election.cc:304] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:50.802361 2788 raft_consensus.cc:2802] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:50.802553 2788 raft_consensus.cc:695] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
W20250623 08:04:50.803306 32589 consensus_peers.cc:487] T 2b405f87bb1e4d0686a1f627835d901f P 125745849172449ba4be8244d61fdcb3 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b405f87bb1e4d0686a1f627835d901f. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:50.802918 2788 consensus_queue.cc:237] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.805028 32368 raft_consensus.cc:2466] T f080a682e532406a83712b6f3e6d2f7b P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.802358 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:50.806097 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30dfa986dfd44658a0ffb07796030520" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.806174 32510 raft_consensus.cc:2391] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:50.806253 32642 raft_consensus.cc:3058] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.801695 952 ts_tablet_manager.cc:1918] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:50.806957 952 log.cc:1199] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/89bb7ea31b4d4ce7a3c063853142fbb2
I20250623 08:04:50.801891 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f080a682e532406a83712b6f3e6d2f7b" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.807606 32641 raft_consensus.cc:3058] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.809146 32642 raft_consensus.cc:2466] T 30dfa986dfd44658a0ffb07796030520 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.810575 32641 raft_consensus.cc:2466] T f080a682e532406a83712b6f3e6d2f7b P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.812322 952 ts_tablet_manager.cc:1939] T 89bb7ea31b4d4ce7a3c063853142fbb2 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:50.828390 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcfe68d4e6ae470d816ac4c494dcaa51" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.828682 32642 raft_consensus.cc:3058] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.829085 342 tablet_bootstrap.cc:492] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:50.829221 342 ts_tablet_manager.cc:1397] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.003s
I20250623 08:04:50.829885 342 raft_consensus.cc:357] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.830039 342 raft_consensus.cc:383] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.830083 342 raft_consensus.cc:738] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.830272 342 consensus_queue.cc:260] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.830883 342 ts_tablet_manager.cc:1428] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:50.831139 342 tablet_bootstrap.cc:492] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:50.831892 32642 raft_consensus.cc:2466] T dcfe68d4e6ae470d816ac4c494dcaa51 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.832377 342 tablet_bootstrap.cc:654] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:50.832463 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 62) took 1342 ms. Trace:
I20250623 08:04:50.834679 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.833312 952 rpcz_store.cc:276] 0623 08:04:49.489542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.489579 (+ 37us) service_pool.cc:225] Handling call
0623 08:04:50.696648 (+1207069us) ts_tablet_manager.cc:1145] Deleting tablet 89bb7ea31b4d4ce7a3c063853142fbb2
0623 08:04:50.696658 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.746997 (+ 50339us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.800691 (+ 53694us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:50.832215 (+ 31524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206870}
I20250623 08:04:50.835477 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 89bb7ea31b4d4ce7a3c063853142fbb2 (table default.Table_28d324870b1049289c32fbfcf8c44ebf [id=cf51a9276ee0490c920bebfef8fbfaf8]) successfully deleted
I20250623 08:04:50.837742 342 tablet_bootstrap.cc:492] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:50.837858 342 ts_tablet_manager.cc:1397] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250623 08:04:50.838213 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 64) took 1348 ms. Trace:
I20250623 08:04:50.838711 952 rpcz_store.cc:276] 0623 08:04:49.489751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.489870 (+ 119us) service_pool.cc:225] Handling call
0623 08:04:50.836823 (+1346953us) ts_tablet_manager.cc:1145] Deleting tablet 3136ec837bea4619ae3995ceec9b24d6
0623 08:04:50.836840 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.837706 (+ 866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346557}
W20250623 08:04:50.839701 1213 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 3136ec837bea4619ae3995ceec9b24d6 because the tablet was not found. No further retry: Not found: Tablet not found: 3136ec837bea4619ae3995ceec9b24d6
I20250623 08:04:50.840418 952 tablet_replica.cc:331] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:50.840963 952 raft_consensus.cc:2241] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:50.838394 342 raft_consensus.cc:357] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.841830 342 raft_consensus.cc:383] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.842236 342 raft_consensus.cc:738] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.842696 1957 raft_consensus.cc:491] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.842876 1957 raft_consensus.cc:513] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:50.843381 1957 leader_election.cc:290] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.843178 342 consensus_queue.cc:260] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.850425 342 ts_tablet_manager.cc:1428] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20250623 08:04:50.851081 342 tablet_bootstrap.cc:492] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:50.854351 952 raft_consensus.cc:2270] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:50.854997 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b8cfcbce6a472ab2daae28226cf766" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.855150 32648 raft_consensus.cc:3058] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.857828 32648 raft_consensus.cc:2466] T 38b8cfcbce6a472ab2daae28226cf766 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.858811 32655 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "492ffaeaaa314dcfb7512d8f23946a67" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.858960 32655 raft_consensus.cc:3058] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.861497 32655 raft_consensus.cc:2466] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.865146 952 ts_tablet_manager.cc:1905] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:50.870087 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eebc32fdfa44abcbd6f04a636d02c7c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.870343 32645 raft_consensus.cc:3058] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.875514 32645 raft_consensus.cc:2466] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.879328 2558 raft_consensus.cc:491] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.879447 2558 raft_consensus.cc:513] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.880076 2558 leader_election.cc:290] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.883056 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee82db25aa544a21a1419e081196f179" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.883229 32656 raft_consensus.cc:3058] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.883374 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa604f500984e3b969e15592cdbed6f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.886073 32656 raft_consensus.cc:2466] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.886188 342 tablet_bootstrap.cc:654] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:50.886970 32589 leader_election.cc:343] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.890087 32645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36d21fe528f5410eacd655bf15306b98" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.890057 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9a0c57133884896b03e9c0a8afb8bde" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.890275 32510 raft_consensus.cc:2466] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:50.890326 32645 raft_consensus.cc:3058] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.891417 2788 raft_consensus.cc:491] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.891655 2788 raft_consensus.cc:513] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.892006 2849 tablet.cc:1722] T f217a8eb12014fcf89b9df4a2cddf5f6 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.892282 2788 leader_election.cc:290] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.893080 32645 raft_consensus.cc:2466] T 36d21fe528f5410eacd655bf15306b98 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:50.899315 32656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.899607 32656 raft_consensus.cc:2391] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:50.902006 2879 tablet.cc:1722] T ecc73f52f0bc43c5bbdca2b89deecb21 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.902681 2880 tablet.cc:1722] T 89a24019b797467e857b744d701fb1ed P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.902793 2849 tablet.cc:1722] T baccec88aefb4056be9578bca13f1591 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.903937 2881 tablet.cc:1722] T 077bb090b3ea4d029463304d7a6feee5 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.908835 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:50.909122 32642 raft_consensus.cc:2391] T f0e24d8d890d4d1db051f172b3ec96ec P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 7351aa31800245659edc9837e27daffd in this term.
I20250623 08:04:50.909502 2558 raft_consensus.cc:491] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.909621 2558 raft_consensus.cc:513] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.909837 2849 tablet.cc:1722] T 74da35cd34c14284af004001b25aa531 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.910151 2558 leader_election.cc:290] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.910881 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054f406c1c34f5d92edeae1b9243777" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:50.911372 32589 leader_election.cc:343] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.916693 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:50.916913 32510 raft_consensus.cc:2391] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:50.921895 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:50.922135 32514 raft_consensus.cc:2391] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:50.924162 2879 tablet.cc:1722] T fd7f3d2e67e84bfcbd45a21658f69e7b P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.926450 2881 tablet.cc:1722] T 42c861e29e1e4e549021dae7255a856c P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.926771 2788 raft_consensus.cc:491] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.927146 2788 raft_consensus.cc:513] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.927862 342 tablet_bootstrap.cc:492] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:50.928213 2788 leader_election.cc:290] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.929519 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41b15c89a744a698ca5eee2f143b4bc" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:50.929682 32514 raft_consensus.cc:3058] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:50.931303 339 tablet_bootstrap.cc:492] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:50.931859 339 ts_tablet_manager.cc:1397] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.158s user 0.006s sys 0.008s
I20250623 08:04:50.932312 32514 raft_consensus.cc:2466] T e41b15c89a744a698ca5eee2f143b4bc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:50.932693 339 raft_consensus.cc:357] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.932940 339 raft_consensus.cc:383] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.933058 339 raft_consensus.cc:738] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.933321 339 consensus_queue.cc:260] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.929805 342 ts_tablet_manager.cc:1397] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.006s
I20250623 08:04:50.934244 2880 tablet.cc:1722] T b0d1ec9d56ba40128325a50f204fe920 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.935550 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e27da2a90c1b43deb3dc90834f4e312e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:50.935747 32510 raft_consensus.cc:2391] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 125745849172449ba4be8244d61fdcb3 in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:50.940299 2878 tablet.cc:1722] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.940521 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9604d4890c940d8992eaf30993bf622" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.941623 342 raft_consensus.cc:357] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.943408 342 raft_consensus.cc:383] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.944013 342 raft_consensus.cc:738] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.944694 2879 tablet.cc:1722] T 16bc969d09f74b0986070a504f9f670e P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.946310 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c529e826fa84a919c018887620e4023" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.947001 32510 raft_consensus.cc:2466] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:50.946424 2881 tablet.cc:1722] T 2e898ede64e641519c8ee3ea1853a4b0 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.946199 342 consensus_queue.cc:260] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.950675 343 ts_tablet_manager.cc:1397] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.383s user 0.024s sys 0.004s
I20250623 08:04:50.951347 343 raft_consensus.cc:357] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.951514 343 raft_consensus.cc:383] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:50.951565 343 raft_consensus.cc:738] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:50.952131 342 ts_tablet_manager.cc:1428] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20250623 08:04:50.954020 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aface2a39d764a6cb6e4965867ab7b38" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.954182 32510 raft_consensus.cc:2466] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 0.
I20250623 08:04:50.955250 342 tablet_bootstrap.cc:492] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:50.956769 2878 tablet.cc:1722] T 23928d64bcef4c06af81698e688b7f94 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:50.959241 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1aa604f500984e3b969e15592cdbed6f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.961123 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3054f406c1c34f5d92edeae1b9243777" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:50.961364 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 74da35cd34c14284af004001b25aa531 (table default.Table_4adeab2edc614899ad6cadb6d86fef97 [id=0c6452b72af345349d656a6cbcdfd491]). Expected version 1 got 0
I20250623 08:04:50.961866 1248 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_8f3f2b27e51a4cea833fbe48bdc7eeec [id=670bededffc8499e8731b15f7e03380c]). Expected version 2 got 0
I20250623 08:04:50.962592 1235 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 42c861e29e1e4e549021dae7255a856c
I20250623 08:04:50.963251 1235 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:57678 (request call id 57) took 3123 ms. Trace:
I20250623 08:04:50.963389 1235 rpcz_store.cc:276] 0623 08:04:47.839317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:50.108322 (+2269005us) service_pool.cc:225] Handling call
0623 08:04:50.135968 (+ 27646us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d9306 00007f79d9223956 00007f79d9223685 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:50.148096 (+ 12128us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:50.148290 (+ 194us) catalog_manager.cc:2790] Modifying in-memory table state
0623 08:04:50.148389 (+ 99us) catalog_manager.cc:2799] Locking tablets
0623 08:04:50.169900 (+ 21511us) catalog_manager.cc:2813] Removing table and tablets from system table
0623 08:04:50.182396 (+ 12496us) catalog_manager.cc:2831] Removing table from by-name map
0623 08:04:50.182501 (+ 105us) catalog_manager.cc:2847] Committing in-memory state
0623 08:04:50.962528 (+780027us) catalog_manager.cc:2851] Aborting table tasks
0623 08:04:50.963175 (+ 647us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:50.171534 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:50.171638 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:50.171643 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:50.171646 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:50.171751 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:50.171763 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:50.171771 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:50.171872 (+ 101us) tablet.cc:699] Row locks acquired
0623 08:04:50.171875 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:50.171907 (+ 32us) write_op.cc:270] Start()
0623 08:04:50.171939 (+ 32us) write_op.cc:276] Timestamp: P: 1750665890171858 usec, L: 0
0623 08:04:50.171943 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:50.172117 (+ 174us) log.cc:844] Serialized 790 byte log entry
0623 08:04:50.181460 (+ 9343us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.181551 (+ 91us) write_op.cc:301] APPLY: starting
0623 08:04:50.181563 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:50.181604 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:50.181641 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:50.181706 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:50.181742 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0623 08:04:50.181749 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:50.181808 (+ 59us) log.cc:844] Serialized 29 byte log entry
0623 08:04:50.181858 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.181874 (+ 16us) write_op.cc:454] Released schema lock
0623 08:04:50.181879 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:50.181942 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.181946 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":62931200,"child_traces":[["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":382,"apply.run_wall_time_us":382,"num_ops":2,"prepare.queue_time_us":769,"prepare.run_cpu_time_us":649,"prepare.run_wall_time_us":650,"replication_time_us":9492}]]}
I20250623 08:04:50.966358 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet 42c861e29e1e4e549021dae7255a856c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:50 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:50.962345 1226 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: default.Table_bb325e4f89c84cf0ae7aa94b67c08817 [id=2979e4296458439ea5753b6ba67634ac] alter complete (version 1)
I20250623 08:04:50.975031 2880 tablet.cc:1722] T 37b38cb54d6c45308358f556a692eabe P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:50.974699 1823 raft_consensus.cc:491] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.975546 1823 raft_consensus.cc:513] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.976142 1823 leader_election.cc:290] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:50.976436 1227 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57678 (request call id 47) took 4109 ms. Trace:
I20250623 08:04:50.976541 1227 rpcz_store.cc:276] 0623 08:04:46.867133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.673723 (+2806590us) service_pool.cc:225] Handling call
0623 08:04:49.676824 (+ 3101us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:49.691049 (+ 14225us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:49.691234 (+ 185us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:49.700649 (+ 9415us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:49.760902 (+ 60253us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:50.976374 (+1215472us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.705973 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.706067 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.706072 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:49.706075 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:49.706149 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.706158 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.706163 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:49.706207 (+ 44us) tablet.cc:699] Row locks acquired
0623 08:04:49.706209 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:49.706233 (+ 24us) write_op.cc:270] Start()
0623 08:04:49.706258 (+ 25us) write_op.cc:276] Timestamp: P: 1750665889706187 usec, L: 0
0623 08:04:49.706262 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.706411 (+ 149us) log.cc:844] Serialized 405 byte log entry
0623 08:04:49.759946 (+ 53535us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.760181 (+ 235us) write_op.cc:301] APPLY: starting
0623 08:04:49.760198 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.760242 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.760293 (+ 51us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.760351 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.760385 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:49.760393 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:49.760486 (+ 93us) log.cc:844] Serialized 23 byte log entry
0623 08:04:49.760730 (+ 244us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.760747 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:49.760753 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.760817 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.760820 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19,"child_traces":[["op",{"apply.queue_time_us":152,"apply.run_cpu_time_us":459,"apply.run_wall_time_us":634,"num_ops":1,"prepare.queue_time_us":4819,"prepare.run_cpu_time_us":500,"prepare.run_wall_time_us":500,"replication_time_us":53651}]]}
I20250623 08:04:50.981827 2718 raft_consensus.cc:491] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:50.981956 2718 raft_consensus.cc:513] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:50.982532 2718 leader_election.cc:290] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:50.982728 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet a383d10d53b04d5699dd3c4bad8878c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:50.983302 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8145567741f34d9aa70c5b7b86f7e1c6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:50.983637 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet d19d468ce13d4108a9badc36fc503306 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:34076
W20250623 08:04:50.983788 32589 leader_election.cc:343] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:50.995311 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet a34da529ffde4595a23fc6596b30e3e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1cee3de5953244f98428747bea912f77) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:50.997117 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet db81250d1e164a7d94affd764fc0ff0c with delete_type TABLET_DATA_DELETED (replaced by tablet 89a24019b797467e857b744d701fb1ed) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:51.001691 339 ts_tablet_manager.cc:1428] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:51.002140 339 tablet_bootstrap.cc:492] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:51.004971 2849 tablet.cc:1722] T 5f957f5575e9475eb9f639e3ad5cb809 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.008536 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet 886fc0fa39c54bddb25ac01f68d95805 with delete_type TABLET_DATA_DELETED (replaced by tablet ec4d50d5f9394afe97e3b43d65e6894e) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:51.011076 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet f0e24d8d890d4d1db051f172b3ec96ec with delete_type TABLET_DATA_DELETED (replaced by tablet a1cf375214c54f3491098db07faedd9f) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:51.012837 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet 42c861e29e1e4e549021dae7255a856c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:50 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.018342 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:51.019381 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bfa787f828b411ab68d81bb896f7fb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 886fc0fa39c54bddb25ac01f68d95805) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.027098 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet 30dfa986dfd44658a0ffb07796030520 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a0615e6f8c346aba5e5f30362bc6de4) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:51.027799 339 tablet_bootstrap.cc:654] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:51.030861 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet a2a07188a4d340bc9caf010d5202e34d with delete_type TABLET_DATA_DELETED (replaced by tablet b91d3650e56441189c3cb74b01cd7fa7) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.032805 2874 raft_consensus.cc:491] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.032984 2874 raft_consensus.cc:513] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.033607 2874 leader_election.cc:290] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.034268 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 42c861e29e1e4e549021dae7255a856c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:50 UTC) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:51.035411 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet f97f14cbaffb46d681d8b11c0213cae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:37 UTC) from {username='slave'} at 127.0.0.1:42984
W20250623 08:04:51.037685 1226 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ee82db25aa544a21a1419e081196f179 consensus state has no leader
I20250623 08:04:51.040120 339 tablet_bootstrap.cc:492] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:51.040377 339 ts_tablet_manager.cc:1397] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.003s
I20250623 08:04:51.041926 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8145567741f34d9aa70c5b7b86f7e1c6" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
W20250623 08:04:51.042364 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 240043us
I20250623 08:04:51.043241 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30dfa986dfd44658a0ffb07796030520" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.043438 32372 raft_consensus.cc:3058] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.041442 339 raft_consensus.cc:357] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.044064 339 raft_consensus.cc:383] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:51.046198 32372 raft_consensus.cc:2466] T 30dfa986dfd44658a0ffb07796030520 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.051108 2659 ts_tablet_manager.cc:1918] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.058183 342 tablet_bootstrap.cc:654] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:51.062852 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcfe68d4e6ae470d816ac4c494dcaa51" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.063055 32377 raft_consensus.cc:3058] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.065934 32377 raft_consensus.cc:2466] T dcfe68d4e6ae470d816ac4c494dcaa51 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.068459 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:51.069801 952 ts_tablet_manager.cc:1918] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:51.069931 952 log.cc:1199] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9fd478c007914d2cb09d929b50322a64
I20250623 08:04:51.070312 952 ts_tablet_manager.cc:1939] T 9fd478c007914d2cb09d929b50322a64 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:51.070640 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 63) took 1581 ms. Trace:
I20250623 08:04:51.070791 952 rpcz_store.cc:276] 0623 08:04:49.489646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490145 (+ 499us) service_pool.cc:225] Handling call
0623 08:04:50.840276 (+1350131us) ts_tablet_manager.cc:1145] Deleting tablet 9fd478c007914d2cb09d929b50322a64
0623 08:04:50.840292 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.939239 (+ 98947us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.069672 (+130433us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.070620 (+ 948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1349957}
I20250623 08:04:51.071390 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 9fd478c007914d2cb09d929b50322a64 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]) successfully deleted
W20250623 08:04:51.060292 1245 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.2:54519 (request call id 142) took 1781 ms (client timeout 1000 ms). Trace:
W20250623 08:04:51.074550 1245 rpcz_store.cc:269] 0623 08:04:49.278650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.468576 (+189926us) service_pool.cc:225] Handling call
0623 08:04:51.060244 (+1591668us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:50.164208 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:50.164322 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:50.164331 (+ 9us) write_op.cc:435] Acquired schema lock
0623 08:04:50.164334 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:50.164515 (+ 181us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:50.164528 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:50.164535 (+ 7us) tablet.cc:683] Acquiring locks for 4 operations
0623 08:04:50.164637 (+ 102us) tablet.cc:699] Row locks acquired
0623 08:04:50.164639 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:50.164672 (+ 33us) write_op.cc:270] Start()
0623 08:04:50.164703 (+ 31us) write_op.cc:276] Timestamp: P: 1750665890164623 usec, L: 0
0623 08:04:50.164707 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:50.164884 (+ 177us) log.cc:844] Serialized 1650 byte log entry
0623 08:04:50.168469 (+ 3585us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.168581 (+ 112us) write_op.cc:301] APPLY: starting
0623 08:04:50.168595 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:50.168685 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:50.168726 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:50.168816 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:50.168850 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0623 08:04:50.168859 (+ 9us) write_op.cc:312] APPLY: finished
0623 08:04:50.168970 (+ 111us) log.cc:844] Serialized 41 byte log entry
0623 08:04:50.169028 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.169050 (+ 22us) write_op.cc:454] Released schema lock
0623 08:04:50.169057 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:50.169130 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.169133 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":131987,"reported_tablets":72,"child_traces":[["op",{"apply.queue_time_us":65,"apply.run_cpu_time_us":541,"apply.run_wall_time_us":541,"num_ops":4,"prepare.queue_time_us":812,"prepare.run_cpu_time_us":768,"prepare.run_wall_time_us":768,"raft.queue_time_us":371,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":228,"replication_time_us":3735}]]}
I20250623 08:04:51.081753 2891 tablet.cc:1722] T 2e898ede64e641519c8ee3ea1853a4b0 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:51.083374 1229 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a375f8ea383b44a9acd04fc2171d34ae consensus state has no leader
I20250623 08:04:51.094007 1229 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57678 (request call id 56) took 3254 ms. Trace:
I20250623 08:04:51.094144 1229 rpcz_store.cc:276] 0623 08:04:47.839208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.973087 (+2133879us) service_pool.cc:225] Handling call
0623 08:04:49.973431 (+ 344us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:49.973679 (+ 248us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:49.973848 (+ 169us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:49.973909 (+ 61us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:50.108560 (+134651us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:51.093965 (+985405us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.974426 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.974534 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.974539 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:49.974541 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:49.974630 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.974640 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.974648 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:49.974699 (+ 51us) tablet.cc:699] Row locks acquired
0623 08:04:49.974701 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:49.974732 (+ 31us) write_op.cc:270] Start()
0623 08:04:49.974774 (+ 42us) write_op.cc:276] Timestamp: P: 1750665889974703 usec, L: 0
0623 08:04:49.974778 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.974923 (+ 145us) log.cc:844] Serialized 405 byte log entry
0623 08:04:50.104579 (+129656us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.105444 (+ 865us) write_op.cc:301] APPLY: starting
0623 08:04:50.105458 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:50.105493 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:50.105536 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:50.105580 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:50.105616 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:50.105624 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:50.105688 (+ 64us) log.cc:844] Serialized 23 byte log entry
0623 08:04:50.105737 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.105749 (+ 12us) write_op.cc:454] Released schema lock
0623 08:04:50.105754 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:50.105815 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.105819 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":827,"apply.run_cpu_time_us":366,"apply.run_wall_time_us":366,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":561,"prepare.run_wall_time_us":561,"replication_time_us":129783}]]}
I20250623 08:04:51.095245 2896 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9752155c20ee42fbb7ff7b7d5b682444)
I20250623 08:04:51.095477 2896 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31835ec4ad74332a00998d431492466" member_type: VOTER last_known_addr { host: "127.31.69.62" port: 39465 } } peers { permanent_uuid: "9752155c20ee42fbb7ff7b7d5b682444" member_type: VOTER last_known_addr { host: "127.31.69.61" port: 33313 } } peers { permanent_uuid: "6644af0f4b2a4b998282b021ec1dc678" member_type: VOTER last_known_addr { host: "127.31.69.60" port: 39691 } }
I20250623 08:04:51.095775 2891 tablet.cc:1722] T 73453eb1a0394cfe8d49adcc71d4f3e9 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:51.096601 2896 leader_election.cc:290] T 00000000000000000000000000000000 P 6644af0f4b2a4b998282b021ec1dc678 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d31835ec4ad74332a00998d431492466 (127.31.69.62:39465), 9752155c20ee42fbb7ff7b7d5b682444 (127.31.69.61:33313)
W20250623 08:04:51.083724 1242 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d7773d6479742f79c38eca34018f94f consensus state has no leader
I20250623 08:04:51.100091 1242 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:57678 (request call id 58) took 3260 ms. Trace:
I20250623 08:04:51.100222 1242 rpcz_store.cc:276] 0623 08:04:47.839400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:50.110051 (+2270651us) service_pool.cc:225] Handling call
0623 08:04:50.136330 (+ 26279us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23da5a6 00007f79d9225616 00007f79d9225345 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:50.148565 (+ 12235us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0623 08:04:50.148807 (+ 242us) catalog_manager.cc:3683] Apply alter schema
0623 08:04:50.148994 (+ 187us) catalog_manager.cc:3715] Acquired catalog manager lock
0623 08:04:50.149070 (+ 76us) catalog_manager.cc:3873] Updating metadata on disk
0623 08:04:50.154284 (+ 5214us) catalog_manager.cc:3904] Committing alterations to in-memory state
0623 08:04:51.100053 (+945769us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:50.149695 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:50.149797 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:50.149803 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:50.149805 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:50.149927 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:50.149939 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:50.149943 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:50.149999 (+ 56us) tablet.cc:699] Row locks acquired
0623 08:04:50.150002 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:50.150034 (+ 32us) write_op.cc:270] Start()
0623 08:04:50.150065 (+ 31us) write_op.cc:276] Timestamp: P: 1750665890149988 usec, L: 0
0623 08:04:50.150071 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:50.150239 (+ 168us) log.cc:844] Serialized 405 byte log entry
0623 08:04:50.153244 (+ 3005us) op_driver.cc:464] REPLICATION: finished
0623 08:04:50.153507 (+ 263us) write_op.cc:301] APPLY: starting
0623 08:04:50.153525 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:50.153566 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:50.153620 (+ 54us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:50.153673 (+ 53us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:50.153704 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:50.153711 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:50.153812 (+ 101us) log.cc:844] Serialized 23 byte log entry
0623 08:04:50.153879 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.153896 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:50.153902 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:50.153968 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:50.153972 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":59889920,"child_traces":[["op",{"apply.queue_time_us":162,"apply.run_cpu_time_us":458,"apply.run_wall_time_us":458,"num_ops":1,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":648,"prepare.run_wall_time_us":648,"raft.queue_time_us":387,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":270,"replication_time_us":3142}]]}
I20250623 08:04:51.102159 1267 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6644af0f4b2a4b998282b021ec1dc678" candidate_term: 5 candidate_status { last_received { term: 4 index: 819 } } ignore_live_leader: false dest_uuid: "9752155c20ee42fbb7ff7b7d5b682444" is_pre_election: true
I20250623 08:04:51.105515 2892 tablet.cc:1722] T 077bb090b3ea4d029463304d7a6feee5 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:51.102281 1245 master_service.cc:948] ReplaceTablet: received request to replace tablet 0e9bf7cba2244703b7402cb4be1a005e from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:51.108609 2891 tablet.cc:1722] T 23928d64bcef4c06af81698e688b7f94 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:51.109190 952 tablet_replica.cc:331] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:51.110522 32374 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fcd0382328f4b0f9ac510c83eb1f68f" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.111065 2887 tablet_replica.cc:331] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:51.109717 342 tablet_bootstrap.cc:492] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:51.111224 32374 raft_consensus.cc:2391] T 3fcd0382328f4b0f9ac510c83eb1f68f P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:51.111248 32381 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e24d8d890d4d1db051f172b3ec96ec" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.112991 2755 raft_consensus.cc:491] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.113314 2755 raft_consensus.cc:513] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.113734 2890 tablet.cc:1722] T fd7f3d2e67e84bfcbd45a21658f69e7b P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.114603 2755 leader_election.cc:290] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.115043 2889 tablet.cc:1722] T b0d1ec9d56ba40128325a50f204fe920 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250623 08:04:51.115967 1226 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b4e01b430a064740b76e0bceec4f5546 consensus state has no leader
I20250623 08:04:51.115926 2887 raft_consensus.cc:2241] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:51.116209 2887 raft_consensus.cc:2270] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.116356 342 ts_tablet_manager.cc:1397] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.162s user 0.001s sys 0.003s
I20250623 08:04:51.117022 952 raft_consensus.cc:2241] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:51.117532 1242 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57678:
name: "default.Table_7756340f9bec45a280c9543136f9c077"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250623 08:04:51.117844 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38b8cfcbce6a472ab2daae28226cf766" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.117974 32380 raft_consensus.cc:3058] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:51.118144 1242 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7756340f9bec45a280c9543136f9c077 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250623 08:04:51.119360 2891 tablet.cc:1722] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:51.120797 32380 raft_consensus.cc:2466] T 38b8cfcbce6a472ab2daae28226cf766 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.121173 32380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "492ffaeaaa314dcfb7512d8f23946a67" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.121316 32380 raft_consensus.cc:3058] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.123993 32380 raft_consensus.cc:2466] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.124621 2659 log.cc:1199] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/b61b02cf2904404f8c0ffef7d268fefc
I20250623 08:04:51.125289 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "207c75acfed64082a885b47ae5353c9d" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.125346 2659 ts_tablet_manager.cc:1939] T b61b02cf2904404f8c0ffef7d268fefc P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:51.125437 32368 raft_consensus.cc:3058] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.051425 343 consensus_queue.cc:260] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.125808 2891 tablet.cc:1722] T 16bc969d09f74b0986070a504f9f670e P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:51.061765 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet 74c1cdc83ffd4a09a163479ac58fd405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:42984
W20250623 08:04:51.063115 32589 leader_election.cc:343] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 2c5ecd4fa5574a1b90b6e82f89ee88f7
I20250623 08:04:51.065666 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e27da2a90c1b43deb3dc90834f4e312e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:51.126097 32589 leader_election.cc:304] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.126288 32653 raft_consensus.cc:2391] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59f7c614254f42009306b13337f06576 in current term 1: Already voted for candidate 125745849172449ba4be8244d61fdcb3 in this term.
I20250623 08:04:51.068182 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f416cdb596a4ed5a302acd07bb84b9c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
W20250623 08:04:51.126425 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 63328.5us
I20250623 08:04:51.126556 32651 raft_consensus.cc:2466] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:51.102593 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3ad3d261a794d1184061c8e9b384693" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.107537 32630 tablet_service.cc:1515] Processing DeleteTablet for tablet ba5913811e9343389c9ea0ec8b8c05ef with delete_type TABLET_DATA_DELETED (replaced by tablet 2ec871b2878a4b3f929b3774746d7750) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.108031 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 886fc0fa39c54bddb25ac01f68d95805 with delete_type TABLET_DATA_DELETED (replaced by tablet ec4d50d5f9394afe97e3b43d65e6894e) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.126881 32642 raft_consensus.cc:2466] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:51.126920 2517 raft_consensus.cc:2747] T 2c5ecd4fa5574a1b90b6e82f89ee88f7 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:51.127998 32368 raft_consensus.cc:2466] T 207c75acfed64082a885b47ae5353c9d P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.128654 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee82db25aa544a21a1419e081196f179" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.128824 32377 raft_consensus.cc:3058] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.131147 32377 raft_consensus.cc:2466] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.131433 32372 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36d21fe528f5410eacd655bf15306b98" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.131615 32372 raft_consensus.cc:3058] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.132387 343 ts_tablet_manager.cc:1428] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.182s user 0.002s sys 0.000s
I20250623 08:04:51.133256 343 tablet_bootstrap.cc:492] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:51.134200 32375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eebc32fdfa44abcbd6f04a636d02c7c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.134409 32375 raft_consensus.cc:3058] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.136358 2890 tablet.cc:1722] T 37b38cb54d6c45308358f556a692eabe P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.137235 32375 raft_consensus.cc:2466] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.138890 32379 raft_consensus.cc:3058] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.140496 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3ad3d261a794d1184061c8e9b384693" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:51.144110 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f416cdb596a4ed5a302acd07bb84b9c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
W20250623 08:04:51.144371 1226 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.31.69.3:50329 (request call id 151) took 1993 ms (client timeout 1000 ms). Trace:
I20250623 08:04:51.144709 2887 ts_tablet_manager.cc:1905] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.146734 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 89bb7ea31b4d4ce7a3c063853142fbb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:35 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.147321 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet a383d10d53b04d5699dd3c4bad8878c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-23 08:04:39 UTC) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.147666 343 tablet_bootstrap.cc:654] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:51.151789 32589 leader_election.cc:304] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
W20250623 08:04:51.152290 32589 leader_election.cc:343] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: f83f68ebda7e4826bbb17278cd4ac5a8
I20250623 08:04:51.152323 2718 raft_consensus.cc:2802] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:51.152407 32589 leader_election.cc:304] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.152441 2718 raft_consensus.cc:491] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:51.152510 2718 raft_consensus.cc:3058] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.152606 32377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e27da2a90c1b43deb3dc90834f4e312e" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.152877 32372 raft_consensus.cc:2466] T 36d21fe528f5410eacd655bf15306b98 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.153007 2659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42984 (request call id 53) took 1629 ms. Trace:
I20250623 08:04:51.153100 2659 rpcz_store.cc:276] 0623 08:04:49.523186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.525121 (+ 1935us) service_pool.cc:225] Handling call
0623 08:04:50.767120 (+1241999us) ts_tablet_manager.cc:1145] Deleting tablet b61b02cf2904404f8c0ffef7d268fefc
0623 08:04:50.767134 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:50.959098 (+191964us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.050974 (+ 91876us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.152968 (+101994us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241803}
I20250623 08:04:51.153551 2659 tablet_replica.cc:331] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:51.153848 2659 raft_consensus.cc:2241] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:51.154522 2659 raft_consensus.cc:2270] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.147945 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b04e38d3da44301a450fe1fb5e2ada6 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f416cdb596a4ed5a302acd07bb84b9c) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.154982 2558 raft_consensus.cc:2747] T f83f68ebda7e4826bbb17278cd4ac5a8 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:51.155553 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15154c85de7144a9b9e2367c70f57e2e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:51.156777 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "492ffaeaaa314dcfb7512d8f23946a67" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:51.156807 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "492ffaeaaa314dcfb7512d8f23946a67" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.156967 32510 raft_consensus.cc:2391] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:51.156998 32641 raft_consensus.cc:2391] T 492ffaeaaa314dcfb7512d8f23946a67 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:51.157316 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "15154c85de7144a9b9e2367c70f57e2e" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.156488 2718 raft_consensus.cc:513] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.157498 32653 raft_consensus.cc:2391] T 15154c85de7144a9b9e2367c70f57e2e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:51.158036 2718 leader_election.cc:290] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.158104 952 raft_consensus.cc:2270] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.158677 952 ts_tablet_manager.cc:1905] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.163158 342 raft_consensus.cc:357] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.163419 342 raft_consensus.cc:383] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:51.163537 342 raft_consensus.cc:738] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:51.163854 342 consensus_queue.cc:260] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.164652 342 ts_tablet_manager.cc:1428] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.048s user 0.000s sys 0.001s
I20250623 08:04:51.165189 342 tablet_bootstrap.cc:492] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:51.166790 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eebc32fdfa44abcbd6f04a636d02c7c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.167068 32653 raft_consensus.cc:2391] T 4eebc32fdfa44abcbd6f04a636d02c7c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:51.167014 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet b61b02cf2904404f8c0ffef7d268fefc (table default.Table_3d6c2aca7a944301bc67f7901cfef8b3 [id=e1d4219a19fc4fffa81e7693469aebce]) successfully deleted
W20250623 08:04:51.167562 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 102248us
I20250623 08:04:51.167961 1823 consensus_queue.cc:1035] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.168095 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4eebc32fdfa44abcbd6f04a636d02c7c" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:51.168303 32510 raft_consensus.cc:2391] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:51.169849 32453 leader_election.cc:304] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.170274 1823 raft_consensus.cc:2802] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.170411 1823 raft_consensus.cc:695] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:51.170697 1823 consensus_queue.cc:237] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.171581 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee82db25aa544a21a1419e081196f179" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.171828 32653 raft_consensus.cc:2391] T ee82db25aa544a21a1419e081196f179 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:51.173276 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ee82db25aa544a21a1419e081196f179" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:51.173457 32510 raft_consensus.cc:2391] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:51.174925 2755 raft_consensus.cc:491] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.175118 2755 raft_consensus.cc:513] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.175801 2755 leader_election.cc:290] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.176865 32589 leader_election.cc:304] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.177501 2558 raft_consensus.cc:2747] T 207c75acfed64082a885b47ae5353c9d P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.180375 32589 leader_election.cc:304] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.181176 2718 raft_consensus.cc:2802] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.181298 2718 raft_consensus.cc:695] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:51.181620 2718 consensus_queue.cc:237] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:51.182893 2446 consensus_queue.cc:1035] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.188140 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ae0a411a3e54c45b8d5c5b9b0eea762" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.188318 32653 raft_consensus.cc:2466] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:51.189667 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ae0a411a3e54c45b8d5c5b9b0eea762" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:51.193759 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:51.195760 32589 leader_election.cc:304] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:51.196687 32314 consensus_peers.cc:487] T 886fc0fa39c54bddb25ac01f68d95805 P 7351aa31800245659edc9837e27daffd -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 886fc0fa39c54bddb25ac01f68d95805. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:51.197329 2558 raft_consensus.cc:2747] T e27da2a90c1b43deb3dc90834f4e312e P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.197322 32314 leader_election.cc:304] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:51.197611 2849 tablet.cc:1722] T e7b9e843ff1344a1aaacf038f15f64cd P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.197924 32453 leader_election.cc:304] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.198005 2755 raft_consensus.cc:2747] T 492ffaeaaa314dcfb7512d8f23946a67 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.198526 32314 leader_election.cc:304] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:51.199473 2755 raft_consensus.cc:2747] T 15154c85de7144a9b9e2367c70f57e2e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.200520 32314 leader_election.cc:304] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:51.201067 2755 raft_consensus.cc:2747] T 4eebc32fdfa44abcbd6f04a636d02c7c P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.202144 32314 leader_election.cc:304] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:51.203048 2755 raft_consensus.cc:2747] T ee82db25aa544a21a1419e081196f179 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.203823 32314 leader_election.cc:304] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
W20250623 08:04:51.144621 1226 rpcz_store.cc:269] 0623 08:04:49.150672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.284356 (+133684us) service_pool.cc:225] Handling call
0623 08:04:51.144219 (+1859863us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:49.647837 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.647978 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.647986 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:49.647990 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:49.648102 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.648117 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.648124 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:49.648192 (+ 68us) tablet.cc:699] Row locks acquired
0623 08:04:49.648197 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:49.648230 (+ 33us) write_op.cc:270] Start()
0623 08:04:49.648264 (+ 34us) write_op.cc:276] Timestamp: P: 1750665889648186 usec, L: 0
0623 08:04:49.648270 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.648481 (+ 211us) log.cc:844] Serialized 527 byte log entry
0623 08:04:49.652361 (+ 3880us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.652477 (+ 116us) write_op.cc:301] APPLY: starting
0623 08:04:49.652494 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.652549 (+ 55us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.652578 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.652632 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.652672 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:49.652683 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:49.652753 (+ 70us) log.cc:844] Serialized 23 byte log entry
0623 08:04:49.652809 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.652830 (+ 21us) write_op.cc:454] Released schema lock
0623 08:04:49.652838 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.652907 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.652911 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0623 08:04:49.656600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:49.656720 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:49.656728 (+ 8us) write_op.cc:435] Acquired schema lock
0623 08:04:49.656732 (+ 4us) tablet.cc:660] Decoding operations
0623 08:04:49.656830 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:49.656845 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:49.656852 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0623 08:04:49.656915 (+ 63us) tablet.cc:699] Row locks acquired
0623 08:04:49.656918 (+ 3us) write_op.cc:260] PREPARE: finished
0623 08:04:49.656947 (+ 29us) write_op.cc:270] Start()
0623 08:04:49.656979 (+ 32us) write_op.cc:276] Timestamp: P: 1750665889656903 usec, L: 0
0623 08:04:49.656985 (+ 6us) op_driver.cc:348] REPLICATION: starting
0623 08:04:49.657160 (+ 175us) log.cc:844] Serialized 356 byte log entry
0623 08:04:49.660401 (+ 3241us) op_driver.cc:464] REPLICATION: finished
0623 08:04:49.660493 (+ 92us) write_op.cc:301] APPLY: starting
0623 08:04:49.660506 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:49.660553 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:49.660588 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:49.660639 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:49.660683 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:49.660691 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:49.660769 (+ 78us) log.cc:844] Serialized 23 byte log entry
0623 08:04:49.660819 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.660834 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:49.660839 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:49.660904 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:49.660908 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":30636,"reported_tablets":82,"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":425,"apply.run_wall_time_us":425,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":739,"prepare.run_wall_time_us":739,"raft.queue_time_us":528,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":255,"replication_time_us":4057}],["op",{"apply.queue_time_us":48,"apply.run_cpu_time_us":405,"apply.run_wall_time_us":404,"num_ops":1,"prepare.queue_time_us":34,"prepare.run_cpu_time_us":653,"prepare.run_wall_time_us":653,"raft.queue_time_us":373,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":212,"replication_time_us":3388}]]}
W20250623 08:04:51.205168 32589 leader_election.cc:343] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.205277 32589 leader_election.cc:304] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.205631 2517 raft_consensus.cc:2747] T e9604d4890c940d8992eaf30993bf622 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.205858 32589 leader_election.cc:304] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.206041 2558 raft_consensus.cc:2802] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:51.206103 2558 raft_consensus.cc:491] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:51.206153 2558 raft_consensus.cc:3058] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.208096 343 tablet_bootstrap.cc:492] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:51.208241 343 ts_tablet_manager.cc:1397] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.001s
I20250623 08:04:51.208817 2558 raft_consensus.cc:513] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.209064 343 raft_consensus.cc:357] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.209220 343 raft_consensus.cc:383] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:51.209262 343 raft_consensus.cc:738] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:51.209393 2755 raft_consensus.cc:2802] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:51.209317 2558 leader_election.cc:290] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.209504 2755 raft_consensus.cc:491] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:51.209601 2755 raft_consensus.cc:3058] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.212318 2755 raft_consensus.cc:513] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.212924 2755 leader_election.cc:290] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.215420 2820 raft_consensus.cc:2802] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.215566 2820 raft_consensus.cc:695] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:51.215783 2820 consensus_queue.cc:237] T 30dfa986dfd44658a0ffb07796030520 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.221357 32514 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ae0a411a3e54c45b8d5c5b9b0eea762" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:51.221338 32642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ae0a411a3e54c45b8d5c5b9b0eea762" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:51.221544 32642 raft_consensus.cc:3058] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:51.222108 32314 leader_election.cc:343] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:51.222739 32314 leader_election.cc:343] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.224928 32642 raft_consensus.cc:2466] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:51.226203 32453 leader_election.cc:304] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.226243 32314 leader_election.cc:304] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:51.227020 2755 raft_consensus.cc:2802] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.227247 2755 raft_consensus.cc:695] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:51.227697 2439 raft_consensus.cc:2802] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.227540 2755 consensus_queue.cc:237] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.227840 2439 raft_consensus.cc:695] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:51.228062 2439 consensus_queue.cc:237] T dcfe68d4e6ae470d816ac4c494dcaa51 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.228871 2659 ts_tablet_manager.cc:1905] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.233659 342 tablet_bootstrap.cc:654] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:51.233836 1226 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a0615e6f8c346aba5e5f30362bc6de4 from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:51.235433 2849 tablet.cc:1722] T a3576624314d4e85b411aab9109683f7 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.209427 343 consensus_queue.cc:260] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.238816 343 ts_tablet_manager.cc:1428] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.030s user 0.001s sys 0.002s
I20250623 08:04:51.239442 343 tablet_bootstrap.cc:492] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:51.240012 2776 consensus_queue.cc:1035] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.247846 952 ts_tablet_manager.cc:1918] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.247978 952 log.cc:1199] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/6b04e38d3da44301a450fe1fb5e2ada6
I20250623 08:04:51.248389 952 ts_tablet_manager.cc:1939] T 6b04e38d3da44301a450fe1fb5e2ada6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
W20250623 08:04:51.248651 32589 leader_election.cc:343] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 803d3607d75443568b7ddc9f0c0bdc90
I20250623 08:04:51.248798 32589 leader_election.cc:304] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.249194 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 49) took 1760 ms. Trace:
I20250623 08:04:51.249205 32589 leader_election.cc:304] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.249294 952 rpcz_store.cc:276] 0623 08:04:49.488354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490592 (+ 2238us) service_pool.cc:225] Handling call
0623 08:04:51.109135 (+1618543us) ts_tablet_manager.cc:1145] Deleting tablet 6b04e38d3da44301a450fe1fb5e2ada6
0623 08:04:51.109148 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.162589 (+ 53441us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.247740 (+ 85151us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.249176 (+ 1436us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40840,"tablet-delete.queue_time_us":1618392}
I20250623 08:04:51.249660 952 tablet_replica.cc:331] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:51.249759 2901 raft_consensus.cc:2747] T 803d3607d75443568b7ddc9f0c0bdc90 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250623 08:04:51.249861 952 raft_consensus.cc:2241] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:51.249863 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 6b04e38d3da44301a450fe1fb5e2ada6 (table default.Table_4ab4668aa0314e1eac46707fe99c1aad [id=340dd4d8076548bfa7a67eabd7dd5d6e]) successfully deleted
I20250623 08:04:51.250006 2901 raft_consensus.cc:2802] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:51.250026 952 raft_consensus.cc:2270] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.250099 2901 raft_consensus.cc:491] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:51.250173 2901 raft_consensus.cc:3058] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.250542 952 ts_tablet_manager.cc:1905] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.253194 2901 raft_consensus.cc:513] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.258285 952 ts_tablet_manager.cc:1918] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.258421 952 log.cc:1199] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/9bfa787f828b411ab68d81bb896f7fb6
I20250623 08:04:51.258960 952 ts_tablet_manager.cc:1939] T 9bfa787f828b411ab68d81bb896f7fb6 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:51.259752 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 72) took 1769 ms. Trace:
I20250623 08:04:51.259860 952 rpcz_store.cc:276] 0623 08:04:49.490466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490712 (+ 246us) service_pool.cc:225] Handling call
0623 08:04:51.249593 (+1758881us) ts_tablet_manager.cc:1145] Deleting tablet 9bfa787f828b411ab68d81bb896f7fb6
0623 08:04:51.249602 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.255031 (+ 5429us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.258205 (+ 3174us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.259732 (+ 1527us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1758675}
I20250623 08:04:51.260252 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 51) took 1771 ms. Trace:
I20250623 08:04:51.260368 952 rpcz_store.cc:276] 0623 08:04:49.488535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490819 (+ 2284us) service_pool.cc:225] Handling call
0623 08:04:51.260158 (+1769339us) ts_tablet_manager.cc:1145] Deleting tablet bb06a340f82047ac8ec3516edb08f4b6
0623 08:04:51.260168 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.260236 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1769199}
I20250623 08:04:51.260668 952 tablet_replica.cc:331] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:51.261122 952 raft_consensus.cc:2241] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:51.261391 952 raft_consensus.cc:2270] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.261924 952 ts_tablet_manager.cc:1905] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.263234 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 9bfa787f828b411ab68d81bb896f7fb6 (table default.Table_38be2bdc4e8f4cd7be412f6d1b883585 [id=c859f297f038430eb111decb794473b5]) successfully deleted
W20250623 08:04:51.263535 1213 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet bb06a340f82047ac8ec3516edb08f4b6 because the tablet was not found. No further retry: Not found: Tablet not found: bb06a340f82047ac8ec3516edb08f4b6
W20250623 08:04:51.263991 32589 leader_election.cc:343] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.264358 32589 leader_election.cc:304] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.264954 2898 raft_consensus.cc:2747] T 1aa604f500984e3b969e15592cdbed6f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.266958 2914 tablet.cc:1722] T 297bdf6c0e424190a35b55cbeefe7161 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250623 08:04:51.268216 32589 leader_election.cc:343] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.269299 952 ts_tablet_manager.cc:1918] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.269455 952 log.cc:1199] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b720fdea539f423ba2d2aa9f89b09ae9
I20250623 08:04:51.269877 952 ts_tablet_manager.cc:1939] T b720fdea539f423ba2d2aa9f89b09ae9 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:51.270664 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 56) took 1781 ms. Trace:
I20250623 08:04:51.271086 32453 leader_election.cc:304] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.271185 2776 consensus_queue.cc:1035] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.271611 32453 leader_election.cc:304] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.271204 952 rpcz_store.cc:276] 0623 08:04:49.488975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490950 (+ 1975us) service_pool.cc:225] Handling call
0623 08:04:51.260627 (+1769677us) ts_tablet_manager.cc:1145] Deleting tablet b720fdea539f423ba2d2aa9f89b09ae9
0623 08:04:51.260636 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.264964 (+ 4328us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.269221 (+ 4257us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.270645 (+ 1424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1769544}
I20250623 08:04:51.271720 2776 raft_consensus.cc:2802] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.271905 952 tablet_replica.cc:331] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:51.272099 952 raft_consensus.cc:2241] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:51.272214 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b720fdea539f423ba2d2aa9f89b09ae9 (table default.Table_59f2123c149e43569f3e3db991d1d09c [id=107d7d6ec5da4d29a8dbd45ba77eb017]) successfully deleted
I20250623 08:04:51.272399 2439 raft_consensus.cc:2802] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.272470 2776 raft_consensus.cc:695] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:51.272518 2439 raft_consensus.cc:695] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:51.272753 2776 consensus_queue.cc:237] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.273013 32453 leader_election.cc:304] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.273212 32453 leader_election.cc:304] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.273311 1957 raft_consensus.cc:2802] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.273391 1957 raft_consensus.cc:695] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:51.273407 32453 leader_election.cc:304] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.273630 1359 raft_consensus.cc:2802] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.273586 1957 consensus_queue.cc:237] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.273877 952 raft_consensus.cc:2270] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.273972 2720 raft_consensus.cc:2802] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.274066 2720 raft_consensus.cc:695] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:51.274250 2720 consensus_queue.cc:237] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.275628 1359 raft_consensus.cc:695] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:51.275857 1359 consensus_queue.cc:237] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.272734 2439 consensus_queue.cc:237] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.281374 952 ts_tablet_manager.cc:1905] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.292009 952 ts_tablet_manager.cc:1918] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.293707 952 log.cc:1199] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/16331ba6a34942a5a1b4ff4ac7bac55e
I20250623 08:04:51.294706 952 ts_tablet_manager.cc:1939] T 16331ba6a34942a5a1b4ff4ac7bac55e P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:51.295897 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 52) took 1807 ms. Trace:
I20250623 08:04:51.296886 952 rpcz_store.cc:276] 0623 08:04:49.488619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.490991 (+ 2372us) service_pool.cc:225] Handling call
0623 08:04:51.271862 (+1780871us) ts_tablet_manager.cc:1145] Deleting tablet 16331ba6a34942a5a1b4ff4ac7bac55e
0623 08:04:51.271871 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.285548 (+ 13677us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.291913 (+ 6365us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.295875 (+ 3962us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1481,"tablet-delete.queue_time_us":1780732}
I20250623 08:04:51.297864 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 70) took 1807 ms. Trace:
I20250623 08:04:51.298025 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 16331ba6a34942a5a1b4ff4ac7bac55e (table default.Table_4533668b8daa4ff3ad00ccc95e97e032 [id=08d03b1b488c4b52abebcef8b3d0f853]) successfully deleted
I20250623 08:04:51.299715 342 tablet_bootstrap.cc:492] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:51.299858 342 ts_tablet_manager.cc:1397] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.135s user 0.004s sys 0.000s
W20250623 08:04:51.298245 32453 leader_election.cc:343] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.300595 342 raft_consensus.cc:357] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.300767 342 raft_consensus.cc:383] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:51.300825 342 raft_consensus.cc:738] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:51.300915 2918 consensus_queue.cc:1035] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.301023 342 consensus_queue.cc:260] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.301133 2849 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 87) took 1064 ms. Trace:
I20250623 08:04:51.298352 952 rpcz_store.cc:276] 0623 08:04:49.490275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.491111 (+ 836us) service_pool.cc:225] Handling call
0623 08:04:51.297764 (+1806653us) ts_tablet_manager.cc:1145] Deleting tablet 4c9d1c8a67e149478a0d69842a4eb5b2
0623 08:04:51.297773 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.297848 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806520}
W20250623 08:04:51.301347 32453 leader_election.cc:343] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:51.301977 32453 leader_election.cc:343] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.302361 32453 leader_election.cc:304] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.302824 2918 raft_consensus.cc:2802] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.302954 2918 raft_consensus.cc:695] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:51.303133 32642 raft_consensus.cc:1273] T 92005abc61dd40ef9da8124cadc29b6a P 125745849172449ba4be8244d61fdcb3 [term 2 FOLLOWER]: Refusing update from remote peer 7351aa31800245659edc9837e27daffd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250623 08:04:51.303179 2918 consensus_queue.cc:237] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.303332 1957 consensus_queue.cc:1035] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.303681 2755 consensus_queue.cc:1035] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250623 08:04:51.313678 1957 consensus_queue.cc:1035] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.315826 32453 leader_election.cc:304] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.316179 2820 raft_consensus.cc:2747] T 3fcd0382328f4b0f9ac510c83eb1f68f P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.316648 343 tablet_bootstrap.cc:654] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:51.317016 1213 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 4c9d1c8a67e149478a0d69842a4eb5b2 because the tablet was not found. No further retry: Not found: Tablet not found: 4c9d1c8a67e149478a0d69842a4eb5b2
I20250623 08:04:51.318177 2820 consensus_queue.cc:1035] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.318858 2887 ts_tablet_manager.cc:1918] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250623 08:04:51.318478 952 tablet_replica.cc:331] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:51.319922 952 raft_consensus.cc:2241] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:51.301524 2849 rpcz_store.cc:276] 0623 08:04:50.236669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:50.236732 (+ 63us) service_pool.cc:225] Handling call
0623 08:04:51.301087 (+1064355us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:50.237416 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:50.237518 (+ 102us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:50.237538 (+ 20us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:50.237547 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:50.237586 (+ 39us) alter_schema_op.cc:128] START. Timestamp: 1750665890237547
0623 08:04:50.237589 (+ 3us) op_driver.cc:348] REPLICATION: starting
0623 08:04:50.237762 (+ 173us) log.cc:844] Serialized 173 byte log entry
0623 08:04:51.235256 (+997494us) op_driver.cc:464] REPLICATION: finished
0623 08:04:51.235413 (+ 157us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:51.236031 (+ 618us) log.cc:844] Serialized 5 byte log entry
0623 08:04:51.293205 (+ 57174us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.300875 (+ 7670us) log.cc:844] Serialized 14 byte log entry
0623 08:04:51.301012 (+ 137us) alter_schema_op.cc:73] Released schema lock
0623 08:04:51.301017 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":92,"prepare.queue_time_us":239,"prepare.run_cpu_time_us":10237,"prepare.run_wall_time_us":16235,"replication_time_us":997632,"thread_start_us":15808,"threads_started":1,"wal-append.queue_time_us":15426}]]}
I20250623 08:04:51.322043 2901 leader_election.cc:290] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.321683 952 raft_consensus.cc:2270] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.325893 32589 leader_election.cc:304] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
W20250623 08:04:51.326354 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 58141.2us
I20250623 08:04:51.326740 2907 raft_consensus.cc:2747] T 3054f406c1c34f5d92edeae1b9243777 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.327382 952 ts_tablet_manager.cc:1905] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.339378 2659 ts_tablet_manager.cc:1918] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.339581 2659 log.cc:1199] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/c74e8c697d244ac2ac8174bb1fc76e61
I20250623 08:04:51.339949 2659 ts_tablet_manager.cc:1939] T c74e8c697d244ac2ac8174bb1fc76e61 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:51.340274 2659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42984 (request call id 57) took 1814 ms. Trace:
I20250623 08:04:51.340363 2659 rpcz_store.cc:276] 0623 08:04:49.525500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.525749 (+ 249us) service_pool.cc:225] Handling call
0623 08:04:51.153485 (+1627736us) ts_tablet_manager.cc:1145] Deleting tablet c74e8c697d244ac2ac8174bb1fc76e61
0623 08:04:51.153494 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.231909 (+ 78415us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.339262 (+107353us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.340254 (+ 992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627587}
I20250623 08:04:51.340862 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet c74e8c697d244ac2ac8174bb1fc76e61 (table default.Table_5785d39307cb42758ded4062307c55e3 [id=4ceb1e1117f747b3a5424c01cd47bd6b]) successfully deleted
I20250623 08:04:51.343585 32453 leader_election.cc:304] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.344411 2918 raft_consensus.cc:2747] T f0e24d8d890d4d1db051f172b3ec96ec P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.349094 2914 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:34076 (request call id 86) took 1134 ms. Trace:
I20250623 08:04:51.353389 1359 consensus_queue.cc:1035] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.354794 32453 leader_election.cc:304] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59f7c614254f42009306b13337f06576; no voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.355760 2439 raft_consensus.cc:2747] T e27da2a90c1b43deb3dc90834f4e312e P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.356808 32453 leader_election.cc:304] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.357810 1242 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:57678 (request call id 59) took 3363 ms. Trace:
I20250623 08:04:51.357949 1242 rpcz_store.cc:276] 0623 08:04:47.994389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:51.100863 (+3106474us) service_pool.cc:225] Handling call
0623 08:04:51.117476 (+ 16613us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23d8313 00007f79d9221c96 00007f79d92219c5 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:51.118251 (+ 775us) catalog_manager.cc:2076] Acquired catalog manager lock
0623 08:04:51.118717 (+ 466us) catalog_manager.cc:2122] Created new table and tablet info
0623 08:04:51.357535 (+238818us) catalog_manager.cc:2182] Wrote table and tablets to system table
0623 08:04:51.357649 (+ 114us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0623 08:04:51.357761 (+ 112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:51.175720 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:51.175868 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:51.176317 (+ 449us) write_op.cc:435] Acquired schema lock
0623 08:04:51.176322 (+ 5us) tablet.cc:660] Decoding operations
0623 08:04:51.176463 (+ 141us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:51.176477 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:51.176485 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:51.176582 (+ 97us) tablet.cc:699] Row locks acquired
0623 08:04:51.176586 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:51.176631 (+ 45us) write_op.cc:270] Start()
0623 08:04:51.176670 (+ 39us) write_op.cc:276] Timestamp: P: 1750665891176584 usec, L: 0
0623 08:04:51.176677 (+ 7us) op_driver.cc:348] REPLICATION: starting
0623 08:04:51.176877 (+ 200us) log.cc:844] Serialized 433 byte log entry
0623 08:04:51.356834 (+179957us) op_driver.cc:464] REPLICATION: finished
0623 08:04:51.356928 (+ 94us) write_op.cc:301] APPLY: starting
0623 08:04:51.356946 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:51.356996 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:51.357034 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:51.357142 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:51.357181 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0623 08:04:51.357192 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:51.357272 (+ 80us) log.cc:844] Serialized 29 byte log entry
0623 08:04:51.357347 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:51.357364 (+ 17us) write_op.cc:454] Released schema lock
0623 08:04:51.357371 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":37544192,"child_traces":[["op",{"apply.queue_time_us":37,"apply.run_cpu_time_us":505,"apply.run_wall_time_us":506,"num_ops":2,"prepare.queue_time_us":56456,"prepare.run_cpu_time_us":831,"prepare.run_wall_time_us":2481,"replication_time_us":180118}]]}
I20250623 08:04:51.358409 1245 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 0e9bf7cba2244703b7402cb4be1a005e
I20250623 08:04:51.358915 1245 catalog_manager.cc:6652] ReplaceTablet: tablet 0e9bf7cba2244703b7402cb4be1a005e deleted and replaced by tablet d65123ebf6f043f8998c2e8472bb6a02
I20250623 08:04:51.359050 1245 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 60) took 3294 ms. Trace:
I20250623 08:04:51.359131 1245 rpcz_store.cc:276] 0623 08:04:48.065029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:51.102212 (+3037183us) service_pool.cc:225] Handling call
0623 08:04:51.131433 (+ 29221us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23e505b 00007f79d922e5d7 00007f79d922e305 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:51.358993 (+227560us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:51.178243 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:51.178359 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:51.178366 (+ 7us) write_op.cc:435] Acquired schema lock
0623 08:04:51.178368 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:51.178461 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:51.178471 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:51.178478 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:51.178565 (+ 87us) tablet.cc:699] Row locks acquired
0623 08:04:51.178569 (+ 4us) write_op.cc:260] PREPARE: finished
0623 08:04:51.178604 (+ 35us) write_op.cc:270] Start()
0623 08:04:51.178641 (+ 37us) write_op.cc:276] Timestamp: P: 1750665891178560 usec, L: 0
0623 08:04:51.178648 (+ 7us) op_driver.cc:348] REPLICATION: starting
0623 08:04:51.182018 (+ 3370us) log.cc:844] Serialized 656 byte log entry
0623 08:04:51.357639 (+175621us) op_driver.cc:464] REPLICATION: finished
0623 08:04:51.357711 (+ 72us) write_op.cc:301] APPLY: starting
0623 08:04:51.357722 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:51.357756 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:51.357794 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:51.357900 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:51.357956 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:51.357967 (+ 11us) write_op.cc:312] APPLY: finished
0623 08:04:51.358025 (+ 58us) log.cc:844] Serialized 29 byte log entry
0623 08:04:51.358076 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:51.358090 (+ 14us) write_op.cc:454] Released schema lock
0623 08:04:51.358095 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":51749248,"child_traces":[["op",{"apply.queue_time_us":32,"num_ops":2,"prepare.queue_time_us":44798,"prepare.run_cpu_time_us":786,"prepare.run_wall_time_us":4572,"replication_time_us":178969}]]}
I20250623 08:04:51.359861 1245 master_service.cc:948] ReplaceTablet: received request to replace tablet e27da2a90c1b43deb3dc90834f4e312e from {username='slave'} at 127.0.0.1:57678
I20250623 08:04:51.361645 2439 raft_consensus.cc:491] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.361783 2439 raft_consensus.cc:513] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.362336 2439 leader_election.cc:290] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:51.362557 32482 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e9bf7cba2244703b7402cb4be1a005e with delete_type TABLET_DATA_DELETED (replaced by tablet d65123ebf6f043f8998c2e8472bb6a02) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:51.362787 2439 raft_consensus.cc:2802] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:51.362859 2439 raft_consensus.cc:491] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:51.362918 2439 raft_consensus.cc:3058] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.364311 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e9bf7cba2244703b7402cb4be1a005e with delete_type TABLET_DATA_DELETED (replaced by tablet d65123ebf6f043f8998c2e8472bb6a02) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.370805 2439 raft_consensus.cc:513] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:51.372116 32481 tablet_service.cc:1468] Processing CreateTablet for tablet 73a77e4138cf4a50b13282ed0b6b3c42 (DEFAULT_TABLE table=default.Table_5e73ae1fb8be4c9da3c5b9b75af18187 [id=57f0ec89dfb2494895f5fd50be3559c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.372236 2439 leader_election.cc:290] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:51.372555 32481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73a77e4138cf4a50b13282ed0b6b3c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.373898 32453 leader_election.cc:304] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.374588 2439 raft_consensus.cc:2802] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:51.374727 2439 raft_consensus.cc:491] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:51.374877 2439 raft_consensus.cc:3058] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.375029 32482 tablet_service.cc:1468] Processing CreateTablet for tablet 48553a3dbfdf49e994c715cd314d5ae9 (DEFAULT_TABLE table=default.Table_6220ff02a44c42e888ef49108c79ec0f [id=2ec07aa3cac84ac194ae7e882ce0b1fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.375367 32482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48553a3dbfdf49e994c715cd314d5ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.379350 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 48553a3dbfdf49e994c715cd314d5ae9 (DEFAULT_TABLE table=default.Table_6220ff02a44c42e888ef49108c79ec0f [id=2ec07aa3cac84ac194ae7e882ce0b1fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.379738 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48553a3dbfdf49e994c715cd314d5ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.383004 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 73a77e4138cf4a50b13282ed0b6b3c42 (DEFAULT_TABLE table=default.Table_5e73ae1fb8be4c9da3c5b9b75af18187 [id=57f0ec89dfb2494895f5fd50be3559c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.383342 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73a77e4138cf4a50b13282ed0b6b3c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.383380 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 4294979f2ac848b5846d1aa6c879ccad (DEFAULT_TABLE table=default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.383713 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4294979f2ac848b5846d1aa6c879ccad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.386976 32492 tablet_service.cc:1468] Processing CreateTablet for tablet fecb5359884b4dddbb73983c08930314 (DEFAULT_TABLE table=default.Table_9504b9b15ec14b209e2ec56b5e4e92df [id=d241e5eab83243758485d43c14525888]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.387316 32492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecb5359884b4dddbb73983c08930314. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.390985 32498 tablet_service.cc:1468] Processing CreateTablet for tablet bd6fe697a9dd467bb8602b95ace5cf63 (DEFAULT_TABLE table=default.Table_caf786518e614b1cb613c11baa90f3cc [id=fdeaf65ac2ab4edc8d65409cbd8cc188]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.391286 32498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6fe697a9dd467bb8602b95ace5cf63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.395044 32497 tablet_service.cc:1468] Processing CreateTablet for tablet 8843b8907efb4aeaa9e5d14309bdb133 (DEFAULT_TABLE table=default.Table_1026838e4fb1446488330a74d347907f [id=45b8db0b4621404aa128957a28519727]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.395370 32497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8843b8907efb4aeaa9e5d14309bdb133. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.400692 32633 tablet_service.cc:1468] Processing CreateTablet for tablet bd6fe697a9dd467bb8602b95ace5cf63 (DEFAULT_TABLE table=default.Table_caf786518e614b1cb613c11baa90f3cc [id=fdeaf65ac2ab4edc8d65409cbd8cc188]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.400784 32636 tablet_service.cc:1468] Processing CreateTablet for tablet fecb5359884b4dddbb73983c08930314 (DEFAULT_TABLE table=default.Table_9504b9b15ec14b209e2ec56b5e4e92df [id=d241e5eab83243758485d43c14525888]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.401086 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6fe697a9dd467bb8602b95ace5cf63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.401275 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 4294979f2ac848b5846d1aa6c879ccad (DEFAULT_TABLE table=default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.401993 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecb5359884b4dddbb73983c08930314. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.402344 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4294979f2ac848b5846d1aa6c879ccad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.404171 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 8843b8907efb4aeaa9e5d14309bdb133 (DEFAULT_TABLE table=default.Table_1026838e4fb1446488330a74d347907f [id=45b8db0b4621404aa128957a28519727]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.404502 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8843b8907efb4aeaa9e5d14309bdb133. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.420253 32618 tablet_service.cc:1468] Processing CreateTablet for tablet 79104f3256dd42ce8e92adf090b5cc8e (DEFAULT_TABLE table=default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.420706 32618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79104f3256dd42ce8e92adf090b5cc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.421919 32500 tablet_service.cc:1468] Processing CreateTablet for tablet 79104f3256dd42ce8e92adf090b5cc8e (DEFAULT_TABLE table=default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.422278 32500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79104f3256dd42ce8e92adf090b5cc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.423004 952 ts_tablet_manager.cc:1918] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.424073 32633 tablet_service.cc:1468] Processing CreateTablet for tablet d3fe0ac83af54051a2f9a019c2bd3969 (DEFAULT_TABLE table=default.Table_f33a210d2a444f62b3beb1415a1775bf [id=fa8064ab42d14fdab05763b9ad014181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.424458 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3fe0ac83af54051a2f9a019c2bd3969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.427845 2879 tablet.cc:1722] T a3576624314d4e85b411aab9109683f7 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.429124 32494 tablet_service.cc:1468] Processing CreateTablet for tablet d3fe0ac83af54051a2f9a019c2bd3969 (DEFAULT_TABLE table=default.Table_f33a210d2a444f62b3beb1415a1775bf [id=fa8064ab42d14fdab05763b9ad014181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.429508 32494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3fe0ac83af54051a2f9a019c2bd3969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.431049 2880 tablet.cc:1722] T 297bdf6c0e424190a35b55cbeefe7161 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:51.433948 2878 tablet.cc:1722] T e7b9e843ff1344a1aaacf038f15f64cd P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.439280 952 log.cc:1199] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/b61b02cf2904404f8c0ffef7d268fefc
I20250623 08:04:51.444342 952 ts_tablet_manager.cc:1939] T b61b02cf2904404f8c0ffef7d268fefc P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:51.447507 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b7416ad6a244d978457e529cbc789de" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.447693 32648 raft_consensus.cc:2466] T 4b7416ad6a244d978457e529cbc789de P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:51.453207 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f416cdb596a4ed5a302acd07bb84b9c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:51.453382 32648 raft_consensus.cc:3058] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.456019 32648 raft_consensus.cc:2466] T 7f416cdb596a4ed5a302acd07bb84b9c P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.461833 32636 tablet_service.cc:1468] Processing CreateTablet for tablet 9b1b29115fdc43db813a62854726df71 (DEFAULT_TABLE table=default.Table_d0add936203f42e78a69c35025573ed6 [id=c35218d165164ae39b5a5ecee52d513d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.463936 32636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b1b29115fdc43db813a62854726df71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.462296 32630 tablet_service.cc:1468] Processing CreateTablet for tablet 7f005a3106184c809b34c7afa1be6cf0 (DEFAULT_TABLE table=default.Table_a65a0f1004274054826e1df3df7ff595 [id=f75f11b73df74cb2a4119881330d6912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.466181 32630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f005a3106184c809b34c7afa1be6cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.452239 2439 raft_consensus.cc:513] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.467168 2439 leader_election.cc:290] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:51.468426 32485 tablet_service.cc:1468] Processing CreateTablet for tablet 9b1b29115fdc43db813a62854726df71 (DEFAULT_TABLE table=default.Table_d0add936203f42e78a69c35025573ed6 [id=c35218d165164ae39b5a5ecee52d513d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.468811 32485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b1b29115fdc43db813a62854726df71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.475693 32489 tablet_service.cc:1468] Processing CreateTablet for tablet 7f005a3106184c809b34c7afa1be6cf0 (DEFAULT_TABLE table=default.Table_a65a0f1004274054826e1df3df7ff595 [id=f75f11b73df74cb2a4119881330d6912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.476156 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f005a3106184c809b34c7afa1be6cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.477916 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 59) took 1988 ms. Trace:
I20250623 08:04:51.478098 952 rpcz_store.cc:276] 0623 08:04:49.489222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.491161 (+ 1939us) service_pool.cc:225] Handling call
0623 08:04:51.317803 (+1826642us) ts_tablet_manager.cc:1145] Deleting tablet b61b02cf2904404f8c0ffef7d268fefc
0623 08:04:51.317814 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.353926 (+ 36112us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.421425 (+ 67499us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.477896 (+ 56471us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":368,"tablet-delete.queue_time_us":1826346}
I20250623 08:04:51.485308 952 tablet_replica.cc:331] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:51.486706 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet b61b02cf2904404f8c0ffef7d268fefc (table default.Table_3d6c2aca7a944301bc67f7901cfef8b3 [id=e1d4219a19fc4fffa81e7693469aebce]) successfully deleted
I20250623 08:04:51.319267 2887 log.cc:1199] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/42c861e29e1e4e549021dae7255a856c
W20250623 08:04:51.321321 32314 consensus_peers.cc:487] T 92005abc61dd40ef9da8124cadc29b6a P 7351aa31800245659edc9837e27daffd -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92005abc61dd40ef9da8124cadc29b6a. This is attempt 1: this message will repeat every 5th retry.
W20250623 08:04:51.487948 32314 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 166648us
I20250623 08:04:51.492620 952 raft_consensus.cc:2241] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:51.493039 952 raft_consensus.cc:2270] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.496215 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:51.494112 952 ts_tablet_manager.cc:1905] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.500363 342 ts_tablet_manager.cc:1428] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.200s user 0.001s sys 0.001s
I20250623 08:04:51.505692 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9a0c57133884896b03e9c0a8afb8bde" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:51.505890 32510 raft_consensus.cc:3058] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.508663 32510 raft_consensus.cc:2466] T e9a0c57133884896b03e9c0a8afb8bde P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:51.512863 342 tablet_bootstrap.cc:492] T 3bfad68d1f20486997ea1b0a8d97206b P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:51.513955 2887 ts_tablet_manager.cc:1939] T 42c861e29e1e4e549021dae7255a856c P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:51.514380 32378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aface2a39d764a6cb6e4965867ab7b38" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.518945 32379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c529e826fa84a919c018887620e4023" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.526311 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9a0c57133884896b03e9c0a8afb8bde" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.526383 1213 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 42c861e29e1e4e549021dae7255a856c (table default.Table_78e064bbf114486f9cfa40945fb67bf2 [id=2fc0e5899bbb4ee39ef17627ccc7b2b9]) successfully deleted
I20250623 08:04:51.528347 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 0e9bf7cba2244703b7402cb4be1a005e with delete_type TABLET_DATA_DELETED (replaced by tablet d65123ebf6f043f8998c2e8472bb6a02) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:51.528939 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 48553a3dbfdf49e994c715cd314d5ae9 (DEFAULT_TABLE table=default.Table_6220ff02a44c42e888ef49108c79ec0f [id=2ec07aa3cac84ac194ae7e882ce0b1fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.529302 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48553a3dbfdf49e994c715cd314d5ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.529817 32342 tablet_service.cc:1468] Processing CreateTablet for tablet 73a77e4138cf4a50b13282ed0b6b3c42 (DEFAULT_TABLE table=default.Table_5e73ae1fb8be4c9da3c5b9b75af18187 [id=57f0ec89dfb2494895f5fd50be3559c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.530138 32342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73a77e4138cf4a50b13282ed0b6b3c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.534267 2919 raft_consensus.cc:491] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.534390 2919 raft_consensus.cc:513] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.535303 32345 tablet_service.cc:1468] Processing CreateTablet for tablet 4294979f2ac848b5846d1aa6c879ccad (DEFAULT_TABLE table=default.Table_8649c62e786b4a0cab61befe1c6c7ec7 [id=bdca6a821a6b48cbb3366d8912abeeda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.535748 32345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4294979f2ac848b5846d1aa6c879ccad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.539644 32348 tablet_service.cc:1468] Processing CreateTablet for tablet fecb5359884b4dddbb73983c08930314 (DEFAULT_TABLE table=default.Table_9504b9b15ec14b209e2ec56b5e4e92df [id=d241e5eab83243758485d43c14525888]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.540050 32348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fecb5359884b4dddbb73983c08930314. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.543084 32357 tablet_service.cc:1468] Processing CreateTablet for tablet bd6fe697a9dd467bb8602b95ace5cf63 (DEFAULT_TABLE table=default.Table_caf786518e614b1cb613c11baa90f3cc [id=fdeaf65ac2ab4edc8d65409cbd8cc188]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.543517 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6fe697a9dd467bb8602b95ace5cf63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.546211 2887 tablet_replica.cc:331] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: stopping tablet replica
I20250623 08:04:51.553208 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:51.547714 32350 tablet_service.cc:1468] Processing CreateTablet for tablet 8843b8907efb4aeaa9e5d14309bdb133 (DEFAULT_TABLE table=default.Table_1026838e4fb1446488330a74d347907f [id=45b8db0b4621404aa128957a28519727]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.554451 2887 raft_consensus.cc:2241] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:51.561159 32512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c529e826fa84a919c018887620e4023" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:51.561371 32512 raft_consensus.cc:3058] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.561745 2659 tablet_replica.cc:331] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:51.562139 2659 raft_consensus.cc:2241] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:51.562534 2659 raft_consensus.cc:2270] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.570732 343 tablet_bootstrap.cc:492] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:51.571034 343 ts_tablet_manager.cc:1397] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.332s user 0.004s sys 0.002s
I20250623 08:04:51.572537 343 raft_consensus.cc:357] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:51.572743 343 raft_consensus.cc:383] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:51.572916 343 raft_consensus.cc:738] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:51.573141 343 consensus_queue.cc:260] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:51.573726 343 ts_tablet_manager.cc:1428] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250623 08:04:51.574019 343 tablet_bootstrap.cc:492] T a375f8ea383b44a9acd04fc2171d34ae P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:51.575045 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:51.575637 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:51.575867 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:51.576421 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:51.576872 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:51.577136 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:51.577345 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:51.577625 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_8f362905029746c19e0e7ce1433c594d [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 2 got 0
I20250623 08:04:51.577874 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:51.578099 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
W20250623 08:04:51.586774 32589 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.31.69.60:39691, user_credentials={real_user=slave}} blocked reactor thread for 221450us
W20250623 08:04:51.589300 32589 leader_election.cc:343] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:51.589599 32589 leader_election.cc:343] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:51.589830 32589 leader_election.cc:343] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.590636 2878 tablet.cc:1722] T b97f3019b893441bae18146fef094bd3 P 125745849172449ba4be8244d61fdcb3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250623 08:04:51.592602 2613 log.cc:927] Time spent T 00000000000000000000000000000000 P d31835ec4ad74332a00998d431492466: Append to log took a long time: real 0.074s user 0.003s sys 0.000s
I20250623 08:04:51.597536 2919 leader_election.cc:290] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:51.605190 1226 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet 4a0615e6f8c346aba5e5f30362bc6de4
I20250623 08:04:51.605746 1226 catalog_manager.cc:6652] ReplaceTablet: tablet 4a0615e6f8c346aba5e5f30362bc6de4 deleted and replaced by tablet 6cb3ebe8dfea41fdb32a80c04e7ab9fb
I20250623 08:04:51.605870 1226 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 61) took 3469 ms. Trace:
I20250623 08:04:51.605978 1226 rpcz_store.cc:276] 0623 08:04:48.136472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:51.226618 (+3090146us) service_pool.cc:225] Handling call
0623 08:04:51.605837 (+379219us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:51.365854 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:51.365965 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:51.365971 (+ 6us) write_op.cc:435] Acquired schema lock
0623 08:04:51.365973 (+ 2us) tablet.cc:660] Decoding operations
0623 08:04:51.366090 (+ 117us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:51.366101 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:51.366113 (+ 12us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:51.366187 (+ 74us) tablet.cc:699] Row locks acquired
0623 08:04:51.366192 (+ 5us) write_op.cc:260] PREPARE: finished
0623 08:04:51.366233 (+ 41us) write_op.cc:270] Start()
0623 08:04:51.366261 (+ 28us) write_op.cc:276] Timestamp: P: 1750665891366170 usec, L: 0
0623 08:04:51.366265 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:51.463002 (+ 96737us) log.cc:844] Serialized 622 byte log entry
0623 08:04:51.463272 (+ 270us) spinlock_profiling.cc:243] Waited 96.4 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 000000000097041a 00007f79da6a15e2 00007f79dba48255 00007f79dba46d79 00007f79dba4f1a0 00007f79dba4efad 0000000000a751bc 00007f79cecb33ff 00007f79cecb66cb 00007f79cecb6548 0000000000a751bc
0623 08:04:51.604256 (+140984us) op_driver.cc:464] REPLICATION: finished
0623 08:04:51.604446 (+ 190us) write_op.cc:301] APPLY: starting
0623 08:04:51.604464 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:51.604512 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:51.604515 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:51.604621 (+ 106us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:51.604682 (+ 61us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:51.604689 (+ 7us) write_op.cc:312] APPLY: finished
0623 08:04:51.604774 (+ 85us) log.cc:844] Serialized 29 byte log entry
0623 08:04:51.604843 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:51.604863 (+ 20us) write_op.cc:454] Released schema lock
0623 08:04:51.604870 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:51.604943 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:51.604947 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":112,"apply.run_cpu_time_us":493,"apply.run_wall_time_us":493,"num_ops":2,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":2201,"prepare.run_wall_time_us":97453,"replication_time_us":237947,"spinlock_wait_cycles":221785472}]]}
I20250623 08:04:51.614962 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a0615e6f8c346aba5e5f30362bc6de4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cb3ebe8dfea41fdb32a80c04e7ab9fb) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:51.615170 32636 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a0615e6f8c346aba5e5f30362bc6de4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cb3ebe8dfea41fdb32a80c04e7ab9fb) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.618614 2887 raft_consensus.cc:2270] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.548614 32346 tablet_service.cc:1468] Processing CreateTablet for tablet d3fe0ac83af54051a2f9a019c2bd3969 (DEFAULT_TABLE table=default.Table_f33a210d2a444f62b3beb1415a1775bf [id=fa8064ab42d14fdab05763b9ad014181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.549441 32354 tablet_service.cc:1468] Processing CreateTablet for tablet 9b1b29115fdc43db813a62854726df71 (DEFAULT_TABLE table=default.Table_d0add936203f42e78a69c35025573ed6 [id=c35218d165164ae39b5a5ecee52d513d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.548403 32355 tablet_service.cc:1468] Processing CreateTablet for tablet 79104f3256dd42ce8e92adf090b5cc8e (DEFAULT_TABLE table=default.Table_f8d48385e5b1441bb296f68cee25e5b2 [id=c463d3625c314596a05115fa648fa166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.555639 32350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8843b8907efb4aeaa9e5d14309bdb133. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.555291 2890 tablet.cc:1722] T 89a24019b797467e857b744d701fb1ed P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.557509 2892 tablet.cc:1722] T e7b9e843ff1344a1aaacf038f15f64cd P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.556668 32353 tablet_service.cc:1468] Processing CreateTablet for tablet 7f005a3106184c809b34c7afa1be6cf0 (DEFAULT_TABLE table=default.Table_a65a0f1004274054826e1df3df7ff595 [id=f75f11b73df74cb2a4119881330d6912]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:51.564004 32364 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f416cdb596a4ed5a302acd07bb84b9c" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.563812 32366 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b7416ad6a244d978457e529cbc789de" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:51.554939 2891 tablet.cc:1722] T ecc73f52f0bc43c5bbdca2b89deecb21 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.566581 2889 tablet.cc:1722] T a3576624314d4e85b411aab9109683f7 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250623 08:04:51.619887 2887 ts_tablet_manager.cc:1905] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.614789 32345 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a0615e6f8c346aba5e5f30362bc6de4 with delete_type TABLET_DATA_DELETED (replaced by tablet 6cb3ebe8dfea41fdb32a80c04e7ab9fb) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:51.623003 2892 tablet.cc:1722] T 297bdf6c0e424190a35b55cbeefe7161 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250623 08:04:51.354110 2914 rpcz_store.cc:276] 0623 08:04:50.215128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:50.215197 (+ 69us) service_pool.cc:225] Handling call
0623 08:04:51.349076 (+1133879us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:50.215657 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0623 08:04:50.215772 (+ 115us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0623 08:04:50.215782 (+ 10us) alter_schema_op.cc:66] Acquired schema lock
0623 08:04:50.215791 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0623 08:04:50.215835 (+ 44us) alter_schema_op.cc:128] START. Timestamp: 1750665890215798
0623 08:04:50.215840 (+ 5us) op_driver.cc:348] REPLICATION: starting
0623 08:04:50.216026 (+ 186us) log.cc:844] Serialized 173 byte log entry
0623 08:04:51.251332 (+1035306us) op_driver.cc:464] REPLICATION: finished
0623 08:04:51.266937 (+ 15605us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0623 08:04:51.325813 (+ 58876us) log.cc:844] Serialized 5 byte log entry
0623 08:04:51.348721 (+ 22908us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.348990 (+ 269us) log.cc:844] Serialized 14 byte log entry
0623 08:04:51.349034 (+ 44us) alter_schema_op.cc:73] Released schema lock
0623 08:04:51.349040 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":15511,"prepare.queue_time_us":85,"prepare.run_cpu_time_us":11237,"prepare.run_wall_time_us":37939,"replication_time_us":1035456,"thread_start_us":53348,"threads_started":2,"wal-append.queue_time_us":36528}]]}
I20250623 08:04:51.627910 2659 ts_tablet_manager.cc:1905] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.629290 2919 raft_consensus.cc:491] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.629688 2919 raft_consensus.cc:513] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.633620 32428 heartbeater.cc:499] Master 127.31.69.61:33313 was elected leader, sending a full tablet report...
W20250623 08:04:51.638041 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 47920.1us
I20250623 08:04:51.646006 2890 tablet.cc:1722] T b97f3019b893441bae18146fef094bd3 P 7351aa31800245659edc9837e27daffd: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250623 08:04:51.647292 32589 leader_election.cc:343] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.647523 32589 leader_election.cc:304] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 125745849172449ba4be8244d61fdcb3; no voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.647823 2718 raft_consensus.cc:2747] T 8145567741f34d9aa70c5b7b86f7e1c6 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:51.654187 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:51.655304 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:51.655622 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 1cee3de5953244f98428747bea912f77 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]). Expected version 1 got 0
I20250623 08:04:51.657047 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:51.657610 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
I20250623 08:04:51.657939 1227 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:51.660444 2919 leader_election.cc:290] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:51.660503 32518 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "aface2a39d764a6cb6e4965867ab7b38" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:51.660691 32518 raft_consensus.cc:3058] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.660707 32512 raft_consensus.cc:2466] T 1c529e826fa84a919c018887620e4023 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:51.666038 343 tablet_bootstrap.cc:654] T a375f8ea383b44a9acd04fc2171d34ae P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:51.675134 32353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f005a3106184c809b34c7afa1be6cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.678043 2887 ts_tablet_manager.cc:1918] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.678623 2887 log.cc:1199] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-0/wal/wals/0e9bf7cba2244703b7402cb4be1a005e
I20250623 08:04:51.678867 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79104f3256dd42ce8e92adf090b5cc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.681749 32518 raft_consensus.cc:2466] T aface2a39d764a6cb6e4965867ab7b38 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125745849172449ba4be8244d61fdcb3 in term 1.
I20250623 08:04:51.682854 32346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3fe0ac83af54051a2f9a019c2bd3969. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.687265 32354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b1b29115fdc43db813a62854726df71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:51.702805 342 tablet_bootstrap.cc:654] T 3bfad68d1f20486997ea1b0a8d97206b P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:51.708339 32653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3ad3d261a794d1184061c8e9b384693" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:51.708539 32653 raft_consensus.cc:3058] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.712062 32653 raft_consensus.cc:2466] T a3ad3d261a794d1184061c8e9b384693 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 1.
I20250623 08:04:51.716125 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3ad3d261a794d1184061c8e9b384693" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd"
I20250623 08:04:51.719285 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2c990297f554adbb2d97f6309758940" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.719478 32651 raft_consensus.cc:2466] T c2c990297f554adbb2d97f6309758940 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:51.719980 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2c990297f554adbb2d97f6309758940" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:51.735880 2718 raft_consensus.cc:491] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.736052 2718 raft_consensus.cc:513] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.736726 2718 leader_election.cc:290] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.741173 2919 raft_consensus.cc:491] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.741307 2919 raft_consensus.cc:513] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.745361 2919 leader_election.cc:290] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:51.745625 32569 tablet.cc:2378] T ba5913811e9343389c9ea0ec8b8c05ef P 59f7c614254f42009306b13337f06576: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250623 08:04:51.747870 339 raft_consensus.cc:738] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:51.748584 343 tablet_bootstrap.cc:492] T a375f8ea383b44a9acd04fc2171d34ae P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:51.753826 952 ts_tablet_manager.cc:1918] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.754855 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f8f8a132e6245f9a04b5bb6f2253c76" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.755465 32651 raft_consensus.cc:2466] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:51.755931 343 ts_tablet_manager.cc:1397] T a375f8ea383b44a9acd04fc2171d34ae P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.182s user 0.003s sys 0.001s
I20250623 08:04:51.756916 343 raft_consensus.cc:357] T a375f8ea383b44a9acd04fc2171d34ae P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.757156 343 raft_consensus.cc:383] T a375f8ea383b44a9acd04fc2171d34ae P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:51.757252 343 raft_consensus.cc:738] T a375f8ea383b44a9acd04fc2171d34ae P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:51.757508 343 consensus_queue.cc:260] T a375f8ea383b44a9acd04fc2171d34ae P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.758172 343 ts_tablet_manager.cc:1428] T a375f8ea383b44a9acd04fc2171d34ae P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:51.758931 343 tablet_bootstrap.cc:492] T 0749cb735acf4cb393ad78a2cd0c0d60 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:51.765218 342 tablet_bootstrap.cc:492] T 3bfad68d1f20486997ea1b0a8d97206b P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:51.771148 952 log.cc:1199] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/59015204545e436cb558fe4465a67c97
I20250623 08:04:51.771867 952 ts_tablet_manager.cc:1939] T 59015204545e436cb558fe4465a67c97 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:51.771373 342 ts_tablet_manager.cc:1397] T 3bfad68d1f20486997ea1b0a8d97206b P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.264s user 0.006s sys 0.000s
I20250623 08:04:51.777415 342 raft_consensus.cc:357] T 3bfad68d1f20486997ea1b0a8d97206b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.777698 342 raft_consensus.cc:383] T 3bfad68d1f20486997ea1b0a8d97206b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:51.777804 342 raft_consensus.cc:738] T 3bfad68d1f20486997ea1b0a8d97206b P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:51.778081 342 consensus_queue.cc:260] T 3bfad68d1f20486997ea1b0a8d97206b P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.779282 342 ts_tablet_manager.cc:1428] T 3bfad68d1f20486997ea1b0a8d97206b P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250623 08:04:51.779582 342 tablet_bootstrap.cc:492] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:51.781797 342 tablet_bootstrap.cc:654] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:51.783071 339 consensus_queue.cc:260] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.783994 339 ts_tablet_manager.cc:1428] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.743s user 0.021s sys 0.001s
W20250623 08:04:51.784117 339 ts_tablet_manager.cc:1460] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd: Tablet startup took 743ms
W20250623 08:04:51.784245 339 ts_tablet_manager.cc:1462] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd: Trace:
0623 08:04:38.123030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:38.123087 (+ 57us) service_pool.cc:225] Handling call
0623 08:04:38.147248 (+ 24161us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0623 08:04:38.147283 (+ 35us) ts_tablet_manager.cc:676] Creating new metadata...
0623 08:04:38.150594 (+ 3311us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:38.168305 (+ 17711us) tablet_replica.cc:206] Creating consensus instance
0623 08:04:38.168492 (+ 187us) inbound_call.cc:173] Queueing success response
0623 08:04:51.002001 (+12833509us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0623 08:04:51.040541 (+ 38540us) ts_tablet_manager.cc:1429] Starting tablet replica
0623 08:04:51.040743 (+ 202us) tablet_replica.cc:260] Starting instrumentation
0623 08:04:51.040868 (+ 125us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":155,"raft.queue_time_us":41,"raft.run_cpu_time_us":41,"raft.run_wall_time_us":42,"tablet-open.queue_time_us":12833540}
I20250623 08:04:51.784878 339 tablet_bootstrap.cc:492] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:51.786538 339 tablet_bootstrap.cc:654] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:51.787523 2887 ts_tablet_manager.cc:1939] T 0e9bf7cba2244703b7402cb4be1a005e P 7351aa31800245659edc9837e27daffd: Deleting consensus metadata
I20250623 08:04:51.751703 2874 consensus_queue.cc:1035] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.788259 1213 catalog_manager.cc:4928] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): tablet 0e9bf7cba2244703b7402cb4be1a005e (table default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c]) successfully deleted
I20250623 08:04:51.790889 32369 raft_consensus.cc:2466] T c2c990297f554adbb2d97f6309758940 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:51.751703 2755 consensus_queue.cc:1035] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250623 08:04:51.756675 32368 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f8f8a132e6245f9a04b5bb6f2253c76" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:51.760082 32364 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4bc93afd147458b833a026e9d0966b8" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:51.806643 2659 ts_tablet_manager.cc:1918] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:51.807199 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 67) took 2317 ms. Trace:
I20250623 08:04:51.811404 2659 log.cc:1199] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/4b11777e0bd04e61a5bc8bb6ec4d97c3
W20250623 08:04:51.816998 1214 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 88f2a63b5a734c8f9b7a53361590245d consensus state has no leader
W20250623 08:04:51.817371 1213 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet 4a0615e6f8c346aba5e5f30362bc6de4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a0615e6f8c346aba5e5f30362bc6de4 already in progress: creating tablet
I20250623 08:04:51.817597 2659 ts_tablet_manager.cc:1939] T 4b11777e0bd04e61a5bc8bb6ec4d97c3 P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
W20250623 08:04:51.816879 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 50882.4us
I20250623 08:04:51.817035 952 rpcz_store.cc:276] 0623 08:04:49.490017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.495710 (+ 5693us) service_pool.cc:225] Handling call
0623 08:04:51.485227 (+1989517us) ts_tablet_manager.cc:1145] Deleting tablet 59015204545e436cb558fe4465a67c97
0623 08:04:51.485237 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.643215 (+157978us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.753702 (+110487us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.807176 (+ 53474us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6789,"tablet-delete.queue_time_us":1989312}
I20250623 08:04:51.828176 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 59015204545e436cb558fe4465a67c97 (table default.Table_094f2e2ebe104774b6a713a5bbb7fd2f [id=9979c6d8878443c09e6ff9f4aea65d62]) successfully deleted
I20250623 08:04:51.828691 952 tablet_replica.cc:331] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:51.830173 2659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42984 (request call id 58) took 2298 ms. Trace:
I20250623 08:04:51.830948 952 raft_consensus.cc:2241] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:51.832217 952 raft_consensus.cc:2270] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250623 08:04:51.831957 32453 consensus_peers.cc:487] T 0e9bf7cba2244703b7402cb4be1a005e P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:51.847491 2914 tablet.cc:1722] T 2e898ede64e641519c8ee3ea1853a4b0 P 59f7c614254f42009306b13337f06576: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250623 08:04:51.762178 32589 leader_election.cc:304] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.851976 342 tablet_bootstrap.cc:492] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:51.855557 342 ts_tablet_manager.cc:1397] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20250623 08:04:51.862990 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:51.863354 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:51.863814 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 5f957f5575e9475eb9f639e3ad5cb809 (table default.Table_91d43acff7904e08ac276de95b652a5c [id=4547bcef8b5d4d25acf8de6cbba4f7bc]). Expected version 1 got 0
I20250623 08:04:51.864200 1235 catalog_manager.cc:5582] T e13dc85076b34903b35310b14dc31085 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:51.864495 2755 consensus_queue.cc:1035] T a2a07188a4d340bc9caf010d5202e34d P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.864851 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:51.865084 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:51.865383 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 11bb55088ab24b94af3e509810890247 (table default.Table_e4c31249050c4016891036cf9507b607 [id=9a7a76e61e1b43508b6f6080f5163417]). Expected version 1 got 0
I20250623 08:04:51.865571 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:51.865984 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:51.866317 1235 catalog_manager.cc:5582] T 0c337a37fcb141bca87212317d750de8 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:51.866873 1245 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9752155c20ee42fbb7ff7b7d5b682444: Sending DeleteTablet for 3 replicas of tablet e27da2a90c1b43deb3dc90834f4e312e
I20250623 08:04:51.867533 1245 catalog_manager.cc:6652] ReplaceTablet: tablet e27da2a90c1b43deb3dc90834f4e312e deleted and replaced by tablet 278a3a95c8b442fe95b6afe0cd504dc7
I20250623 08:04:51.867656 1245 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:57678 (request call id 62) took 3669 ms. Trace:
I20250623 08:04:51.867794 32355 tablet_service.cc:1515] Processing DeleteTablet for tablet e27da2a90c1b43deb3dc90834f4e312e with delete_type TABLET_DATA_DELETED (replaced by tablet 278a3a95c8b442fe95b6afe0cd504dc7) from {username='slave'} at 127.0.0.1:41574
I20250623 08:04:51.867767 1245 rpcz_store.cc:276] 0623 08:04:48.198686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:51.359765 (+3161079us) service_pool.cc:225] Handling call
0623 08:04:51.463264 (+103499us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x616000041254. stack: 00007f79cec6dd56 00007f79cec6db10 00007f79cdc6429d 0000000000970a84 0000000000970a4d 00007f79dda0afcd 00007f79dda0b0c3 00007f79da6cf1c1 00007f79e224f2ed 00007f79e23e505b 00007f79d922e5d7 00007f79d922e305 00007f79d5a1193e 00007f79d5a1081e 00007f79d5a16a55 00007f79d5a1737f
0623 08:04:51.867625 (+404361us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0623 08:04:51.853820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0623 08:04:51.853918 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0623 08:04:51.853923 (+ 5us) write_op.cc:435] Acquired schema lock
0623 08:04:51.853926 (+ 3us) tablet.cc:660] Decoding operations
0623 08:04:51.854019 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0623 08:04:51.854027 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0623 08:04:51.854031 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0623 08:04:51.854097 (+ 66us) tablet.cc:699] Row locks acquired
0623 08:04:51.854099 (+ 2us) write_op.cc:260] PREPARE: finished
0623 08:04:51.854129 (+ 30us) write_op.cc:270] Start()
0623 08:04:51.854155 (+ 26us) write_op.cc:276] Timestamp: P: 1750665891854067 usec, L: 0
0623 08:04:51.854159 (+ 4us) op_driver.cc:348] REPLICATION: starting
0623 08:04:51.854324 (+ 165us) log.cc:844] Serialized 622 byte log entry
0623 08:04:51.860774 (+ 6450us) op_driver.cc:464] REPLICATION: finished
0623 08:04:51.860874 (+ 100us) write_op.cc:301] APPLY: starting
0623 08:04:51.860887 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0623 08:04:51.860922 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0623 08:04:51.860949 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0623 08:04:51.861040 (+ 91us) tablet.cc:1381] finished ApplyRowOperation cycle
0623 08:04:51.861107 (+ 67us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0623 08:04:51.861115 (+ 8us) write_op.cc:312] APPLY: finished
0623 08:04:51.861188 (+ 73us) log.cc:844] Serialized 29 byte log entry
0623 08:04:51.861243 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:51.861258 (+ 15us) write_op.cc:454] Released schema lock
0623 08:04:51.861263 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0623 08:04:51.861335 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0623 08:04:51.861340 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5233,"spinlock_wait_cycles":237127936,"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":454,"apply.run_wall_time_us":453,"mutex_wait_us":81542,"num_ops":2,"prepare.queue_time_us":82026,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":579,"replication_time_us":6590}]]}
I20250623 08:04:51.868198 32633 tablet_service.cc:1515] Processing DeleteTablet for tablet e27da2a90c1b43deb3dc90834f4e312e with delete_type TABLET_DATA_DELETED (replaced by tablet 278a3a95c8b442fe95b6afe0cd504dc7) from {username='slave'} at 127.0.0.1:42984
I20250623 08:04:51.870891 32489 tablet_service.cc:1515] Processing DeleteTablet for tablet e27da2a90c1b43deb3dc90834f4e312e with delete_type TABLET_DATA_DELETED (replaced by tablet 278a3a95c8b442fe95b6afe0cd504dc7) from {username='slave'} at 127.0.0.1:34076
I20250623 08:04:51.872084 2901 raft_consensus.cc:2802] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.872192 2901 raft_consensus.cc:695] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:51.872427 2901 consensus_queue.cc:237] T e9a0c57133884896b03e9c0a8afb8bde P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.874405 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:51.878088 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
W20250623 08:04:51.879258 32589 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.69.2:40197, user_credentials={real_user=slave}} blocked reactor thread for 117092us
W20250623 08:04:51.880474 32589 consensus_peers.cc:487] T a34da529ffde4595a23fc6596b30e3e9 P 125745849172449ba4be8244d61fdcb3 -> Peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Couldn't send request to peer 59f7c614254f42009306b13337f06576. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a34da529ffde4595a23fc6596b30e3e9. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:51.880729 32589 leader_election.cc:304] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.881034 2901 raft_consensus.cc:2802] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.881274 2901 raft_consensus.cc:695] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:51.881588 2901 consensus_queue.cc:237] T 1c529e826fa84a919c018887620e4023 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.881916 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet baccec88aefb4056be9578bca13f1591 (table default.Table_d0e3d64801be4520a9c8d9c6b99f78ef [id=0a1bbca429e6427581026c4600755351]). Expected version 1 got 0
I20250623 08:04:51.884258 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
W20250623 08:04:51.886562 1213 catalog_manager.cc:4908] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): delete failed for tablet e27da2a90c1b43deb3dc90834f4e312e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e27da2a90c1b43deb3dc90834f4e312e already in progress: creating tablet
I20250623 08:04:51.886950 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:51.889909 32589 leader_election.cc:304] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:51.890237 2898 raft_consensus.cc:2802] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:51.890352 2898 raft_consensus.cc:695] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Running, Role: LEADER
I20250623 08:04:51.890599 2898 consensus_queue.cc:237] T aface2a39d764a6cb6e4965867ab7b38 P 125745849172449ba4be8244d61fdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.889164 1235 catalog_manager.cc:5582] T 15154c85de7144a9b9e2367c70f57e2e P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:51.892540 2874 consensus_queue.cc:1035] T f0e24d8d890d4d1db051f172b3ec96ec P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:51.897271 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
W20250623 08:04:51.902716 32589 leader_election.cc:343] T d4bc93afd147458b833a026e9d0966b8 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.903463 339 tablet_bootstrap.cc:492] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:51.903967 339 ts_tablet_manager.cc:1397] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.000s
I20250623 08:04:51.904325 2659 rpcz_store.cc:276] 0623 08:04:49.531729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.531783 (+ 54us) service_pool.cc:225] Handling call
0623 08:04:51.561692 (+2029909us) ts_tablet_manager.cc:1145] Deleting tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3
0623 08:04:51.561704 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:51.667859 (+106155us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.806518 (+138659us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:51.830147 (+ 23629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2029736}
I20250623 08:04:51.904901 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 4b11777e0bd04e61a5bc8bb6ec4d97c3 (table default.Table_5d6ec4f6805b4e0883b8d0419625d033 [id=545b478879aa4f4cb35393e3065d3ff7]) successfully deleted
I20250623 08:04:51.907650 339 raft_consensus.cc:357] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.911674 339 raft_consensus.cc:383] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:51.914006 339 raft_consensus.cc:738] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
W20250623 08:04:51.919063 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 66045.2us
I20250623 08:04:51.919093 342 raft_consensus.cc:357] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.919778 342 raft_consensus.cc:383] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:51.920007 342 raft_consensus.cc:738] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:51.920660 342 consensus_queue.cc:260] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.921509 2659 tablet_replica.cc:331] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:51.921725 2659 raft_consensus.cc:2241] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:51.921989 2659 raft_consensus.cc:2270] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:51.923928 342 ts_tablet_manager.cc:1428] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.067s user 0.004s sys 0.000s
I20250623 08:04:51.924919 2874 raft_consensus.cc:491] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.925045 2874 raft_consensus.cc:513] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.925650 2874 leader_election.cc:290] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.926126 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4e01b430a064740b76e0bceec4f5546" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:51.926294 32641 raft_consensus.cc:2466] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:51.926443 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4e01b430a064740b76e0bceec4f5546" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:51.926616 32314 leader_election.cc:304] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:51.926931 2874 raft_consensus.cc:2802] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:51.927006 2874 raft_consensus.cc:491] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:51.927079 2874 raft_consensus.cc:3058] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
W20250623 08:04:51.927228 32314 leader_election.cc:343] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.925705 339 consensus_queue.cc:260] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.929845 2874 raft_consensus.cc:513] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.928411 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:51.930454 2874 leader_election.cc:290] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.930826 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4e01b430a064740b76e0bceec4f5546" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:51.930964 32641 raft_consensus.cc:3058] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:51.931412 32510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4e01b430a064740b76e0bceec4f5546" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
W20250623 08:04:51.931954 32314 leader_election.cc:343] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.838152 343 tablet_bootstrap.cc:654] T 0749cb735acf4cb393ad78a2cd0c0d60 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:51.933537 32641 raft_consensus.cc:2466] T b4e01b430a064740b76e0bceec4f5546 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:51.946273 342 tablet_bootstrap.cc:492] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:51.953207 2718 raft_consensus.cc:491] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:51.953338 2718 raft_consensus.cc:513] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:51.953917 2718 leader_election.cc:290] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:51.962244 339 ts_tablet_manager.cc:1428] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.057s user 0.005s sys 0.000s
I20250623 08:04:51.983482 339 tablet_bootstrap.cc:492] T 1cee3de5953244f98428747bea912f77 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:51.992417 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
W20250623 08:04:51.993539 32453 leader_election.cc:343] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:51.993745 32453 leader_election.cc:343] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:51.994616 1235 catalog_manager.cc:5582] T 28d79c96bd1c4ddfb4a1dd8a9d83c3fe P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:51.995743 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:51.996198 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:51.996543 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:51.998091 2659 ts_tablet_manager.cc:1905] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:51.999325 1235 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_012962846ea34a55be604efd6b4d286b [id=670bededffc8499e8731b15f7e03380c]). Expected version 3 got 0
I20250623 08:04:52.002811 342 tablet_bootstrap.cc:654] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.007273 339 tablet_bootstrap.cc:654] T 1cee3de5953244f98428747bea912f77 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.031538 343 tablet_bootstrap.cc:492] T 0749cb735acf4cb393ad78a2cd0c0d60 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:52.031783 343 ts_tablet_manager.cc:1397] T 0749cb735acf4cb393ad78a2cd0c0d60 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.273s user 0.004s sys 0.000s
I20250623 08:04:52.032550 343 raft_consensus.cc:357] T 0749cb735acf4cb393ad78a2cd0c0d60 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.032680 32314 leader_election.cc:304] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:52.032846 343 raft_consensus.cc:383] T 0749cb735acf4cb393ad78a2cd0c0d60 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.032954 343 raft_consensus.cc:738] T 0749cb735acf4cb393ad78a2cd0c0d60 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.033125 2874 raft_consensus.cc:2802] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:52.033252 2874 raft_consensus.cc:695] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:52.033275 343 consensus_queue.cc:260] T 0749cb735acf4cb393ad78a2cd0c0d60 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.033483 2874 consensus_queue.cc:237] T b4e01b430a064740b76e0bceec4f5546 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.034032 343 ts_tablet_manager.cc:1428] T 0749cb735acf4cb393ad78a2cd0c0d60 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:52.034402 343 tablet_bootstrap.cc:492] T 115763354f294954a260b0aa5c66a146 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:52.036273 343 tablet_bootstrap.cc:654] T 115763354f294954a260b0aa5c66a146 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:52.037972 32453 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.69.1:34877, user_credentials={real_user=slave}} blocked reactor thread for 44073.8us
I20250623 08:04:52.042426 339 tablet_bootstrap.cc:492] T 1cee3de5953244f98428747bea912f77 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:52.042604 339 ts_tablet_manager.cc:1397] T 1cee3de5953244f98428747bea912f77 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.000s
I20250623 08:04:52.043296 339 raft_consensus.cc:357] T 1cee3de5953244f98428747bea912f77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.043474 339 raft_consensus.cc:383] T 1cee3de5953244f98428747bea912f77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.043555 339 raft_consensus.cc:738] T 1cee3de5953244f98428747bea912f77 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.043766 339 consensus_queue.cc:260] T 1cee3de5953244f98428747bea912f77 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.044279 339 ts_tablet_manager.cc:1428] T 1cee3de5953244f98428747bea912f77 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250623 08:04:52.044643 339 tablet_bootstrap.cc:492] T 7f416cdb596a4ed5a302acd07bb84b9c P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:52.045913 343 tablet_bootstrap.cc:492] T 115763354f294954a260b0aa5c66a146 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:52.045948 339 tablet_bootstrap.cc:654] T 7f416cdb596a4ed5a302acd07bb84b9c P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.046118 343 ts_tablet_manager.cc:1397] T 115763354f294954a260b0aa5c66a146 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20250623 08:04:52.047073 343 raft_consensus.cc:357] T 115763354f294954a260b0aa5c66a146 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.047303 343 raft_consensus.cc:383] T 115763354f294954a260b0aa5c66a146 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.047406 343 raft_consensus.cc:738] T 115763354f294954a260b0aa5c66a146 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.047676 343 consensus_queue.cc:260] T 115763354f294954a260b0aa5c66a146 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.048328 343 ts_tablet_manager.cc:1428] T 115763354f294954a260b0aa5c66a146 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:52.048630 343 tablet_bootstrap.cc:492] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:52.048985 2919 consensus_queue.cc:1035] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:52.050338 32453 leader_election.cc:343] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:52.051549 343 tablet_bootstrap.cc:654] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.055174 32364 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d7773d6479742f79c38eca34018f94f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:52.056775 343 tablet_bootstrap.cc:492] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:52.057559 339 tablet_bootstrap.cc:492] T 7f416cdb596a4ed5a302acd07bb84b9c P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:52.057757 343 ts_tablet_manager.cc:1397] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20250623 08:04:52.058449 343 raft_consensus.cc:357] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.058779 339 ts_tablet_manager.cc:1397] T 7f416cdb596a4ed5a302acd07bb84b9c P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250623 08:04:52.059509 339 raft_consensus.cc:357] T 7f416cdb596a4ed5a302acd07bb84b9c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.059731 343 raft_consensus.cc:383] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.059857 343 raft_consensus.cc:738] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.060107 343 consensus_queue.cc:260] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.060318 339 raft_consensus.cc:383] T 7f416cdb596a4ed5a302acd07bb84b9c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.060485 339 raft_consensus.cc:738] T 7f416cdb596a4ed5a302acd07bb84b9c P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.061036 343 ts_tablet_manager.cc:1428] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:52.061283 339 consensus_queue.cc:260] T 7f416cdb596a4ed5a302acd07bb84b9c P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.061831 339 ts_tablet_manager.cc:1428] T 7f416cdb596a4ed5a302acd07bb84b9c P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250623 08:04:52.062155 343 tablet_bootstrap.cc:492] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:52.063567 339 tablet_bootstrap.cc:492] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:52.064549 2958 consensus_queue.cc:1035] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:52.065191 343 tablet_bootstrap.cc:654] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.065558 339 tablet_bootstrap.cc:654] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.068260 2958 raft_consensus.cc:491] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:52.068399 2958 raft_consensus.cc:513] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.068894 2958 leader_election.cc:290] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
W20250623 08:04:52.070379 32589 leader_election.cc:343] T 3d7773d6479742f79c38eca34018f94f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:52.072492 2718 consensus_queue.cc:1035] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:52.080646 339 tablet_bootstrap.cc:492] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:52.080844 339 ts_tablet_manager.cc:1397] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.002s
I20250623 08:04:52.081601 339 raft_consensus.cc:357] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.081840 339 raft_consensus.cc:383] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.081945 339 raft_consensus.cc:738] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.082268 339 consensus_queue.cc:260] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.084146 339 ts_tablet_manager.cc:1428] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250623 08:04:52.084474 339 tablet_bootstrap.cc:492] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:52.088465 339 tablet_bootstrap.cc:654] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:52.093148 32453 consensus_peers.cc:487] T 42c861e29e1e4e549021dae7255a856c P 59f7c614254f42009306b13337f06576 -> Peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Couldn't send request to peer 7351aa31800245659edc9837e27daffd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42c861e29e1e4e549021dae7255a856c. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:52.094805 342 tablet_bootstrap.cc:492] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:52.094946 342 ts_tablet_manager.cc:1397] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.166s user 0.002s sys 0.009s
I20250623 08:04:52.096047 342 raft_consensus.cc:357] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.096328 342 raft_consensus.cc:383] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.096406 342 raft_consensus.cc:738] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.096899 342 consensus_queue.cc:260] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.097644 342 ts_tablet_manager.cc:1428] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250623 08:04:52.097975 342 tablet_bootstrap.cc:492] T b4e01b430a064740b76e0bceec4f5546 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:52.099992 342 tablet_bootstrap.cc:654] T b4e01b430a064740b76e0bceec4f5546 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.094870 32453 leader_election.cc:304] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f7c614254f42009306b13337f06576, 7351aa31800245659edc9837e27daffd; no voters:
I20250623 08:04:52.103229 343 tablet_bootstrap.cc:492] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:52.103360 343 ts_tablet_manager.cc:1397] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.009s
I20250623 08:04:52.103999 343 raft_consensus.cc:357] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.104156 343 raft_consensus.cc:383] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.104202 343 raft_consensus.cc:738] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.104818 343 consensus_queue.cc:260] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.105188 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:52.105748 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 16bc969d09f74b0986070a504f9f670e (table default.Table_483c90d6ac4445e482e0eedd47c52bb8 [id=e6e52d5a7c6c4bef978d652a1ca81e47]). Expected version 2 got 0
I20250623 08:04:52.106012 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:52.106287 343 ts_tablet_manager.cc:1428] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250623 08:04:52.106495 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 37b38cb54d6c45308358f556a692eabe (table default.Table_a774d79724e6498fa40937c65bf75b80 [id=5d3fcb492ac246419c43b0618b4d797d]). Expected version 1 got 0
I20250623 08:04:52.106587 343 tablet_bootstrap.cc:492] T b300341952d34600ba745245f51482d4 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:52.106765 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:52.106998 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b0d1ec9d56ba40128325a50f204fe920 (table default.Table_50fb8135a67a4f3cbf9250135b3ded35 [id=3e41f8259c5648c59b388ea9f4de99d5]). Expected version 1 got 0
I20250623 08:04:52.107213 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet fd7f3d2e67e84bfcbd45a21658f69e7b (table default.Table_06be929600454320935c1f072c6629dc [id=d5366c10bade4a2b93f0a2b2524517fa]). Expected version 1 got 0
I20250623 08:04:52.107492 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 73453eb1a0394cfe8d49adcc71d4f3e9 (table default.Table_45c6b0778f6d4874afc4875669e6b179 [id=bee83212a54d4711ad16ad829e99470b]). Expected version 2 got 0
I20250623 08:04:52.108053 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
I20250623 08:04:52.108135 343 tablet_bootstrap.cc:654] T b300341952d34600ba745245f51482d4 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.108310 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:52.108879 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 5f1fb8231e8e45208ac4f412134d8696 (table default.Table_1428aeeb452648f0beaadc1b2f78c83b [id=e25e0edebbac486785025ef14bcf5352]). Expected version 1 got 0
I20250623 08:04:52.109396 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e602d391526d4b0eb0dbf5189afbe075 (table default.Table_50f04b0656c9469fb7bb4beee4b9268c [id=426f83a7100640699646c6a02f20c6cc]). Expected version 2 got 0
I20250623 08:04:52.109871 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 51fce46407ac4e57a74ad696e85daac7 (table default.Table_34bf849295714bdab1ec5c40496934af [id=25eb4e3f3a6e42e988d2a3613ef4575f]). Expected version 3 got 0
I20250623 08:04:52.110357 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 23928d64bcef4c06af81698e688b7f94 (table default.Table_4248729f2b014a7cbe5da578d4a1e5af [id=ea13915f7ffd4e9c9116fb0b12fe0351]). Expected version 2 got 0
I20250623 08:04:52.110590 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:52.111310 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 3d7773d6479742f79c38eca34018f94f (table default.Table_9eed3165e01347d3a7492a52966d849f [id=e85cbbd0d9db4945930b3fca419e3f2c]). Expected version 1 got 0
I20250623 08:04:52.111893 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:52.112293 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 7f890ee0747b4df0b82bc3b396191124 (table default.Table_7ccb5edd6b824f308d349c51d926d963 [id=9f97f6deca194137a353ea221d0e3676]). Expected version 1 got 0
I20250623 08:04:52.112799 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 1cee3de5953244f98428747bea912f77 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]). Expected version 1 got 0
I20250623 08:04:52.113183 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ca5c694856f44201a8aa4314a4f81681 (table default.Table_682a42dbf21d43d49ad9d85b958b121c [id=4f81957f51bd40c192ad7c716cbd893f]). Expected version 1 got 0
I20250623 08:04:52.113374 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 035bb06bc7dd42f681886def9d56eb73 (table default.Table_53879d6f821f4167bf02ffbe967f73f7 [id=b3f407a7a26e4bd5ae9a719a5c4577c0]). Expected version 1 got 0
I20250623 08:04:52.113586 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet ecc73f52f0bc43c5bbdca2b89deecb21 (table default.Table_88ef305b303f49708fc3db029ffb8ace [id=677e05a247254ed8b35326b17b9616e9]). Expected version 1 got 0
I20250623 08:04:52.114006 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:52.114306 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 87648b11b10b45fc9cc8c998bbceae92 (table default.Table_19247d878b0b463aaadf4577f627c2ad [id=ad88a05d1bbb448e94daa1471a373cae]). Expected version 1 got 0
I20250623 08:04:52.115140 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:52.115377 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 89a24019b797467e857b744d701fb1ed (table default.Table_912f5673d23f4f0797a33298858f2789 [id=77e1411772b54c89a9a8435654249287]). Expected version 1 got 0
I20250623 08:04:52.115792 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c50655c06cd94c67a49513c39728eaa9 (table default.Table_e192d00b3ef4472c8dbe2a9fdadc0290 [id=793edd0cb585499197d2f0e57a83be84]). Expected version 1 got 0
I20250623 08:04:52.116326 2918 raft_consensus.cc:2802] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:52.116441 2918 raft_consensus.cc:491] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:52.116528 2918 raft_consensus.cc:3058] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:52.116624 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:52.116966 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_8f362905029746c19e0e7ce1433c594d [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 2 got 0
I20250623 08:04:52.117426 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:52.117643 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a375f8ea383b44a9acd04fc2171d34ae (table default.Table_523e0491c028492db09a84adbbfe5baa [id=e6cd13b44efa4cbbbdd2c14d8ea22267]). Expected version 2 got 0
I20250623 08:04:52.117904 1229 catalog_manager.cc:5582] T 4e3920ac85e54379abbcc42dd80db504 P 7351aa31800245659edc9837e27daffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7351aa31800245659edc9837e27daffd (127.31.69.1). New cstate: current_term: 1 leader_uuid: "7351aa31800245659edc9837e27daffd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:52.118436 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:52.118775 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 077bb090b3ea4d029463304d7a6feee5 (table default.Table_79cb8b58ae4c493db6f796981a67564d [id=387b3fc3d26b457099bcbb3b8ef3ac9b]). Expected version 2 got 0
I20250623 08:04:52.120675 2918 raft_consensus.cc:513] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.121418 2918 leader_election.cc:290] T c2c990297f554adbb2d97f6309758940 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:52.121881 1229 catalog_manager.cc:5674] TS 7351aa31800245659edc9837e27daffd (127.31.69.1:34877) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_012962846ea34a55be604efd6b4d286b [id=670bededffc8499e8731b15f7e03380c]). Expected version 3 got 0
W20250623 08:04:52.122258 32453 leader_election.cc:343] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:52.128623 32427 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:52.129904 952 ts_tablet_manager.cc:1905] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:52.139317 2958 consensus_queue.cc:1035] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:52.147684 1242 catalog_manager.cc:5582] T c8e4cde2af694f07a9176313b4b93a46 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.148528 1242 catalog_manager.cc:5582] T 4eebc32fdfa44abcbd6f04a636d02c7c P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.148855 2659 ts_tablet_manager.cc:1918] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:52.148898 1242 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
I20250623 08:04:52.149595 1242 catalog_manager.cc:5582] T 36d21fe528f5410eacd655bf15306b98 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.150076 1242 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:52.150295 342 tablet_bootstrap.cc:492] T b4e01b430a064740b76e0bceec4f5546 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:52.150302 1242 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:52.150476 342 ts_tablet_manager.cc:1397] T b4e01b430a064740b76e0bceec4f5546 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20250623 08:04:52.150671 1242 catalog_manager.cc:5582] T 492ffaeaaa314dcfb7512d8f23946a67 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.159653 32453 leader_election.cc:304] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:52.159955 2958 raft_consensus.cc:2802] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:52.160048 2958 raft_consensus.cc:491] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:52.160121 2958 raft_consensus.cc:3058] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:52.160292 32453 leader_election.cc:304] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:52.160743 2918 raft_consensus.cc:2802] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:52.160912 2918 raft_consensus.cc:695] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:52.161207 2918 consensus_queue.cc:237] T 7f416cdb596a4ed5a302acd07bb84b9c P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.163564 2958 raft_consensus.cc:513] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.164094 2958 leader_election.cc:290] T 4b7416ad6a244d978457e529cbc789de P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:52.151268 1242 catalog_manager.cc:5582] T 207c75acfed64082a885b47ae5353c9d P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.167407 1242 catalog_manager.cc:5582] T ee82db25aa544a21a1419e081196f179 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.167825 1242 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:52.169651 1242 catalog_manager.cc:5582] T 38b8cfcbce6a472ab2daae28226cf766 P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.170094 1242 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_8f362905029746c19e0e7ce1433c594d [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 2 got 0
I20250623 08:04:52.170272 1242 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
I20250623 08:04:52.170497 1242 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:52.173171 342 raft_consensus.cc:357] T b4e01b430a064740b76e0bceec4f5546 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.173399 342 raft_consensus.cc:383] T b4e01b430a064740b76e0bceec4f5546 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.173496 342 raft_consensus.cc:738] T b4e01b430a064740b76e0bceec4f5546 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.173871 342 consensus_queue.cc:260] T b4e01b430a064740b76e0bceec4f5546 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.174494 342 ts_tablet_manager.cc:1428] T b4e01b430a064740b76e0bceec4f5546 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250623 08:04:52.176692 1242 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_012962846ea34a55be604efd6b4d286b [id=670bededffc8499e8731b15f7e03380c]). Expected version 3 got 2
I20250623 08:04:52.177349 1242 catalog_manager.cc:5582] T 88f2a63b5a734c8f9b7a53361590245d P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.177719 1242 catalog_manager.cc:5674] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:52.179335 1242 catalog_manager.cc:5582] T f080a682e532406a83712b6f3e6d2f7b P 59f7c614254f42009306b13337f06576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59f7c614254f42009306b13337f06576 (127.31.69.2). New cstate: current_term: 1 leader_uuid: "59f7c614254f42009306b13337f06576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.199520 342 tablet_bootstrap.cc:492] T 1cee3de5953244f98428747bea912f77 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:52.201472 342 tablet_bootstrap.cc:654] T 1cee3de5953244f98428747bea912f77 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
W20250623 08:04:52.181141 32453 consensus_peers.cc:487] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 -> Peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Couldn't send request to peer 125745849172449ba4be8244d61fdcb3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c74e8c697d244ac2ac8174bb1fc76e61. This is attempt 1: this message will repeat every 5th retry.
I20250623 08:04:52.203336 32453 leader_election.cc:304] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:52.203653 2962 raft_consensus.cc:2802] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:52.203795 2962 raft_consensus.cc:695] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Becoming Leader. State: Replica: 59f7c614254f42009306b13337f06576, State: Running, Role: LEADER
I20250623 08:04:52.204011 2962 consensus_queue.cc:237] T a3ad3d261a794d1184061c8e9b384693 P 59f7c614254f42009306b13337f06576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.171968 2901 raft_consensus.cc:491] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:52.209472 2901 raft_consensus.cc:513] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.210244 2901 leader_election.cc:290] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:52.191035 2898 raft_consensus.cc:491] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:52.210722 2898 raft_consensus.cc:513] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.211369 2898 leader_election.cc:290] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:52.211606 2659 log.cc:1199] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-2/wal/wals/9dbbdbebdf3b4b50bda31a94fc0de36f
I20250623 08:04:52.212615 2659 ts_tablet_manager.cc:1939] T 9dbbdbebdf3b4b50bda31a94fc0de36f P 125745849172449ba4be8244d61fdcb3: Deleting consensus metadata
I20250623 08:04:52.217260 32489 tablet_service.cc:1468] Processing CreateTablet for tablet b907f5ff29a44cffa45e8b87bd8b782e (DEFAULT_TABLE table=default.Table_7756340f9bec45a280c9543136f9c077 [id=67d4e3277ad8484eacbc19b69e246b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:52.217725 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b907f5ff29a44cffa45e8b87bd8b782e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:52.220150 32364 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f20b23fd6e5c44f58b1fb2bf617c5704" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:52.220530 32371 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13824916241c4cf596119d44d89cd17f" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:52.221876 952 ts_tablet_manager.cc:1918] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:52.223100 952 log.cc:1199] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/74c1cdc83ffd4a09a163479ac58fd405
I20250623 08:04:52.228960 952 ts_tablet_manager.cc:1939] T 74c1cdc83ffd4a09a163479ac58fd405 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:52.232261 339 tablet_bootstrap.cc:492] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:52.232460 339 ts_tablet_manager.cc:1397] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.148s user 0.034s sys 0.030s
I20250623 08:04:52.233114 339 raft_consensus.cc:357] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.233992 343 tablet_bootstrap.cc:492] T b300341952d34600ba745245f51482d4 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:52.234172 343 ts_tablet_manager.cc:1397] T b300341952d34600ba745245f51482d4 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.128s user 0.023s sys 0.036s
I20250623 08:04:52.233615 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 76) took 2644 ms. Trace:
I20250623 08:04:52.235131 343 raft_consensus.cc:357] T b300341952d34600ba745245f51482d4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.235378 343 raft_consensus.cc:383] T b300341952d34600ba745245f51482d4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.235478 343 raft_consensus.cc:738] T b300341952d34600ba745245f51482d4 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.235770 343 consensus_queue.cc:260] T b300341952d34600ba745245f51482d4 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.236341 343 ts_tablet_manager.cc:1428] T b300341952d34600ba745245f51482d4 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250623 08:04:52.233275 339 raft_consensus.cc:383] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.236649 32633 tablet_service.cc:1468] Processing CreateTablet for tablet b907f5ff29a44cffa45e8b87bd8b782e (DEFAULT_TABLE table=default.Table_7756340f9bec45a280c9543136f9c077 [id=67d4e3277ad8484eacbc19b69e246b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:52.236843 339 raft_consensus.cc:738] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.237031 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b907f5ff29a44cffa45e8b87bd8b782e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:52.234949 952 rpcz_store.cc:276] 0623 08:04:49.589124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.591049 (+ 1925us) service_pool.cc:225] Handling call
0623 08:04:51.828637 (+2237588us) ts_tablet_manager.cc:1145] Deleting tablet 74c1cdc83ffd4a09a163479ac58fd405
0623 08:04:51.828649 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:52.177428 (+348779us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:52.220693 (+ 43265us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:52.233399 (+ 12706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2237367}
I20250623 08:04:52.237161 339 consensus_queue.cc:260] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.238273 339 ts_tablet_manager.cc:1428] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250623 08:04:52.238329 952 tablet_replica.cc:331] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:52.238585 339 tablet_bootstrap.cc:492] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:52.234937 32355 tablet_service.cc:1468] Processing CreateTablet for tablet b907f5ff29a44cffa45e8b87bd8b782e (DEFAULT_TABLE table=default.Table_7756340f9bec45a280c9543136f9c077 [id=67d4e3277ad8484eacbc19b69e246b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:52.239162 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b907f5ff29a44cffa45e8b87bd8b782e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:52.239956 952 raft_consensus.cc:2241] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:52.240520 339 tablet_bootstrap.cc:654] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.243031 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet 74c1cdc83ffd4a09a163479ac58fd405 (table default.Table_fbb1fc5910454112b795f72c26f1af5b [id=6d6ee37c928745ef877266573fe021f6]) successfully deleted
I20250623 08:04:52.243507 2898 raft_consensus.cc:491] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:52.243678 2898 raft_consensus.cc:513] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.244383 2898 leader_election.cc:290] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:52.249078 952 raft_consensus.cc:2270] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:52.249563 2659 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42984 (request call id 60) took 2621 ms. Trace:
I20250623 08:04:52.250067 339 tablet_bootstrap.cc:492] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 7351aa31800245659edc9837e27daffd: No bootstrap required, opened a new log
I20250623 08:04:52.250229 339 ts_tablet_manager.cc:1397] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 7351aa31800245659edc9837e27daffd: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20250623 08:04:52.250972 339 raft_consensus.cc:357] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.251127 339 raft_consensus.cc:383] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.251225 339 raft_consensus.cc:738] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351aa31800245659edc9837e27daffd, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.251451 339 consensus_queue.cc:260] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 7351aa31800245659edc9837e27daffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.251953 952 ts_tablet_manager.cc:1905] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:52.251461 2659 rpcz_store.cc:276] 0623 08:04:49.628019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.628068 (+ 49us) service_pool.cc:225] Handling call
0623 08:04:51.921444 (+2293376us) ts_tablet_manager.cc:1145] Deleting tablet 9dbbdbebdf3b4b50bda31a94fc0de36f
0623 08:04:51.921454 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:52.002717 (+ 81263us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:52.148673 (+145956us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:52.249539 (+100866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2293178}
I20250623 08:04:52.254506 32453 leader_election.cc:304] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576; no voters: 7351aa31800245659edc9837e27daffd
I20250623 08:04:52.255201 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec4d50d5f9394afe97e3b43d65e6894e" candidate_uuid: "125745849172449ba4be8244d61fdcb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:52.255834 1213 catalog_manager.cc:4928] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): tablet 9dbbdbebdf3b4b50bda31a94fc0de36f (table default.Table_ff6e3cec929a433b8ac68152b9095dfc [id=ce7434eee7664a8686eb22f28df7f1d8]) successfully deleted
I20250623 08:04:52.256773 2659 tablet_replica.cc:331] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3: stopping tablet replica
I20250623 08:04:52.257566 2659 raft_consensus.cc:2241] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 1 LEADER]: Raft consensus shutting down.
I20250623 08:04:52.257980 339 ts_tablet_manager.cc:1428] T 9bb99da3b8fa4dc6aa7f11a829d3f2d2 P 7351aa31800245659edc9837e27daffd: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250623 08:04:52.257988 2961 raft_consensus.cc:2802] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:52.258100 2961 raft_consensus.cc:491] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:52.258203 2961 raft_consensus.cc:3058] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:52.258355 339 tablet_bootstrap.cc:492] T 3bfad68d1f20486997ea1b0a8d97206b P 7351aa31800245659edc9837e27daffd: Bootstrap starting.
I20250623 08:04:52.259716 339 tablet_bootstrap.cc:654] T 3bfad68d1f20486997ea1b0a8d97206b P 7351aa31800245659edc9837e27daffd: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.261735 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 297bdf6c0e424190a35b55cbeefe7161 (table default.Table_dab86e75c8674afab09def2e63dbfb43 [id=a3e24dd912cb43d19c4eea2b259b3d90]). Expected version 2 got 0
I20250623 08:04:52.265898 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a3576624314d4e85b411aab9109683f7 (table default.Table_3c51bbf186704672adf120f825160733 [id=cb527c3a47d74b459b377b64b23f7675]). Expected version 1 got 0
I20250623 08:04:52.267069 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet b4e01b430a064740b76e0bceec4f5546 (table default.Table_46ce438e1ca54a23abcbf79c87396f71 [id=7a2cd6d0739b48dca9ef91af18569181]). Expected version 1 got 0
I20250623 08:04:52.267336 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 88f2a63b5a734c8f9b7a53361590245d (table default.Table_1b5a331260574f8eb6a61a556aee3a8e [id=6b0ae7e5a9e64bdaa14dd5bf6f5a6451]). Expected version 1 got 0
I20250623 08:04:52.267558 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 0c337a37fcb141bca87212317d750de8 (table default.Table_6ac4b9cba3e1422faf62f4002fb70452 [id=3b957d976c904262895846aaabc9596e]). Expected version 1 got 0
I20250623 08:04:52.267649 2961 raft_consensus.cc:513] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.268316 2961 leader_election.cc:290] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [CANDIDATE]: Term 1 election: Requested vote from peers 7351aa31800245659edc9837e27daffd (127.31.69.1:34877), 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225)
I20250623 08:04:52.268600 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet e7b9e843ff1344a1aaacf038f15f64cd (table default.Table_a96d9f3fd33f485c8a9015f78f39a7e9 [id=e1e321dbfa254d21b11fc31f07d4de07]). Expected version 1 got 0
I20250623 08:04:52.269254 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 2e898ede64e641519c8ee3ea1853a4b0 (table default.Table_012962846ea34a55be604efd6b4d286b [id=670bededffc8499e8731b15f7e03380c]). Expected version 3 got 2
I20250623 08:04:52.269449 32355 tablet_service.cc:1468] Processing CreateTablet for tablet d65123ebf6f043f8998c2e8472bb6a02 (DEFAULT_TABLE table=default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:52.269848 32355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d65123ebf6f043f8998c2e8472bb6a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:52.270004 2446 raft_consensus.cc:491] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:52.270117 2446 raft_consensus.cc:513] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.270674 2446 leader_election.cc:290] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:52.271203 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet a3ad3d261a794d1184061c8e9b384693 (table default.Table_5eda9b192e4f4d66b647be71493a31a5 [id=86475d6fed9c4f8b84556bcc1b1030be]). Expected version 1 got 0
I20250623 08:04:52.271453 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 4eebc32fdfa44abcbd6f04a636d02c7c (table default.Table_faafe8f5734d448da0da414ad0c4e9ac [id=123fe4cae52542ddbde750a8de19320f]). Expected version 1 got 0
W20250623 08:04:52.271872 32314 leader_election.cc:343] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): Not found: Tablet not found: 8911308855cb4414be426dbcec592f4a
I20250623 08:04:52.271896 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet c2c990297f554adbb2d97f6309758940 (table default.Table_8f362905029746c19e0e7ce1433c594d [id=0ea2aee143054b39ae96e30c64a42a50]). Expected version 2 got 0
I20250623 08:04:52.272151 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet b97f3019b893441bae18146fef094bd3 (table default.Table_133d7b35a6ac4c6b8093e39cd1db48db [id=6e032739cfda445a8eb2188ecf94c816]). Expected version 1 got 0
W20250623 08:04:52.272274 32314 leader_election.cc:343] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225): Not found: Tablet not found: 8911308855cb4414be426dbcec592f4a
I20250623 08:04:52.272372 32314 leader_election.cc:304] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7351aa31800245659edc9837e27daffd; no voters: 125745849172449ba4be8244d61fdcb3, 59f7c614254f42009306b13337f06576
I20250623 08:04:52.272475 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 3bfad68d1f20486997ea1b0a8d97206b (table default.Table_b8a2a531e35748bab1f7751ece1f6b06 [id=3c89160b4fb54f649fd3d222f6a6699d]). Expected version 1 got 0
I20250623 08:04:52.272782 2446 raft_consensus.cc:2747] T 8911308855cb4414be426dbcec592f4a P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250623 08:04:52.273002 1226 catalog_manager.cc:5582] T e41b15c89a744a698ca5eee2f143b4bc P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } }
I20250623 08:04:52.273689 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet e41b15c89a744a698ca5eee2f143b4bc (table default.Table_41ce11c5261f4bcfb883fadb408fddc0 [id=2158e340258845d19f99a6300ec4c86d]). Expected version 1 got 0
I20250623 08:04:52.273973 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet ee82db25aa544a21a1419e081196f179 (table default.Table_2ab902dacb4a4c7898a6ab01df011eb4 [id=89148e37f2454772a74b0c88e4164f0b]). Expected version 1 got 0
I20250623 08:04:52.274206 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 1cee3de5953244f98428747bea912f77 (table default.Table_df8f082e8b3c45178294a9ab36df0fdc [id=eb6d9c5745e649978f87c97627ace9bc]). Expected version 1 got 0
I20250623 08:04:52.274731 32489 tablet_service.cc:1468] Processing CreateTablet for tablet d65123ebf6f043f8998c2e8472bb6a02 (DEFAULT_TABLE table=default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:52.274930 1226 catalog_manager.cc:5582] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 125745849172449ba4be8244d61fdcb3 (127.31.69.3). New cstate: current_term: 1 leader_uuid: "125745849172449ba4be8244d61fdcb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20250623 08:04:52.275806 32489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d65123ebf6f043f8998c2e8472bb6a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:52.276310 1226 catalog_manager.cc:5674] TS 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225) does not have the latest schema for tablet 3d7773d6479742f79c38eca34018f94f (table default.Table_9eed3165e01347d3a7492a52966d849f [id=e85cbbd0d9db4945930b3fca419e3f2c]). Expected version 1 got 0
I20250623 08:04:52.276957 342 tablet_bootstrap.cc:492] T 1cee3de5953244f98428747bea912f77 P 59f7c614254f42009306b13337f06576: No bootstrap required, opened a new log
I20250623 08:04:52.278364 342 ts_tablet_manager.cc:1397] T 1cee3de5953244f98428747bea912f77 P 59f7c614254f42009306b13337f06576: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.003s
I20250623 08:04:52.278683 2446 raft_consensus.cc:491] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250623 08:04:52.278831 2446 raft_consensus.cc:513] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.279119 342 raft_consensus.cc:357] T 1cee3de5953244f98428747bea912f77 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.279278 342 raft_consensus.cc:383] T 1cee3de5953244f98428747bea912f77 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.279326 342 raft_consensus.cc:738] T 1cee3de5953244f98428747bea912f77 P 59f7c614254f42009306b13337f06576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f7c614254f42009306b13337f06576, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.279404 2446 leader_election.cc:290] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:52.279537 342 consensus_queue.cc:260] T 1cee3de5953244f98428747bea912f77 P 59f7c614254f42009306b13337f06576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.280051 342 ts_tablet_manager.cc:1428] T 1cee3de5953244f98428747bea912f77 P 59f7c614254f42009306b13337f06576: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250623 08:04:52.280165 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f8f8a132e6245f9a04b5bb6f2253c76" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576" is_pre_election: true
I20250623 08:04:52.280308 342 tablet_bootstrap.cc:492] T 1494f83d13cd49c7a804593f45f97cc6 P 59f7c614254f42009306b13337f06576: Bootstrap starting.
I20250623 08:04:52.280357 32516 raft_consensus.cc:2391] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:52.281742 342 tablet_bootstrap.cc:654] T 1494f83d13cd49c7a804593f45f97cc6 P 59f7c614254f42009306b13337f06576: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.236673 343 tablet_bootstrap.cc:492] T 7f890ee0747b4df0b82bc3b396191124 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:52.283949 2659 raft_consensus.cc:2270] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250623 08:04:52.284453 2659 ts_tablet_manager.cc:1905] T 59015204545e436cb558fe4465a67c97 P 125745849172449ba4be8244d61fdcb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250623 08:04:52.285425 343 tablet_bootstrap.cc:654] T 7f890ee0747b4df0b82bc3b396191124 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.289881 32567 heartbeater.cc:499] Master 127.31.69.62:39465 was elected leader, sending a full tablet report...
I20250623 08:04:52.274372 32633 tablet_service.cc:1468] Processing CreateTablet for tablet d65123ebf6f043f8998c2e8472bb6a02 (DEFAULT_TABLE table=default.Table_7c8c0d6c2a854f93845730b012a19628 [id=ee80d713d08d4d758af2dda7a9de616c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250623 08:04:52.280860 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f8f8a132e6245f9a04b5bb6f2253c76" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:52.294399 32641 raft_consensus.cc:2466] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 0.
I20250623 08:04:52.294979 32633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d65123ebf6f043f8998c2e8472bb6a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250623 08:04:52.294936 32314 leader_election.cc:304] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:52.295390 2446 raft_consensus.cc:2802] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250623 08:04:52.295478 2446 raft_consensus.cc:491] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250623 08:04:52.295557 2446 raft_consensus.cc:3058] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:52.298118 2446 raft_consensus.cc:513] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.298836 343 tablet_bootstrap.cc:492] T 7f890ee0747b4df0b82bc3b396191124 P 125745849172449ba4be8244d61fdcb3: No bootstrap required, opened a new log
I20250623 08:04:52.298959 343 ts_tablet_manager.cc:1397] T 7f890ee0747b4df0b82bc3b396191124 P 125745849172449ba4be8244d61fdcb3: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20250623 08:04:52.299630 343 raft_consensus.cc:357] T 7f890ee0747b4df0b82bc3b396191124 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.299875 343 raft_consensus.cc:383] T 7f890ee0747b4df0b82bc3b396191124 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250623 08:04:52.299968 343 raft_consensus.cc:738] T 7f890ee0747b4df0b82bc3b396191124 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125745849172449ba4be8244d61fdcb3, State: Initialized, Role: FOLLOWER
I20250623 08:04:52.299896 32516 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f8f8a132e6245f9a04b5bb6f2253c76" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f7c614254f42009306b13337f06576"
I20250623 08:04:52.300132 32516 raft_consensus.cc:2391] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7351aa31800245659edc9837e27daffd in current term 1: Already voted for candidate 59f7c614254f42009306b13337f06576 in this term.
I20250623 08:04:52.300177 343 consensus_queue.cc:260] T 7f890ee0747b4df0b82bc3b396191124 P 125745849172449ba4be8244d61fdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } }
I20250623 08:04:52.300666 32641 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f8f8a132e6245f9a04b5bb6f2253c76" candidate_uuid: "7351aa31800245659edc9837e27daffd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3"
I20250623 08:04:52.300808 32641 raft_consensus.cc:3058] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:52.298696 2446 leader_election.cc:290] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Requested vote from peers 125745849172449ba4be8244d61fdcb3 (127.31.69.3:35225), 59f7c614254f42009306b13337f06576 (127.31.69.2:40197)
I20250623 08:04:52.302645 952 ts_tablet_manager.cc:1918] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250623 08:04:52.303292 952 log.cc:1199] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: Deleting WAL directory at /tmp/dist-test-taskJxinPB/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1750665857933363-32020-0/minicluster-data/ts-1/wal/wals/c74e8c697d244ac2ac8174bb1fc76e61
I20250623 08:04:52.303577 32641 raft_consensus.cc:2466] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 125745849172449ba4be8244d61fdcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7351aa31800245659edc9837e27daffd in term 1.
I20250623 08:04:52.304059 32314 leader_election.cc:304] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 125745849172449ba4be8244d61fdcb3, 7351aa31800245659edc9837e27daffd; no voters: 59f7c614254f42009306b13337f06576
I20250623 08:04:52.304105 343 ts_tablet_manager.cc:1428] T 7f890ee0747b4df0b82bc3b396191124 P 125745849172449ba4be8244d61fdcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250623 08:04:52.304401 343 tablet_bootstrap.cc:492] T ca5c694856f44201a8aa4314a4f81681 P 125745849172449ba4be8244d61fdcb3: Bootstrap starting.
I20250623 08:04:52.304515 2446 raft_consensus.cc:2802] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 1 FOLLOWER]: Leader election won for term 1
I20250623 08:04:52.304760 2446 raft_consensus.cc:695] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [term 1 LEADER]: Becoming Leader. State: Replica: 7351aa31800245659edc9837e27daffd, State: Running, Role: LEADER
I20250623 08:04:52.305051 2446 consensus_queue.cc:237] T 8f8f8a132e6245f9a04b5bb6f2253c76 P 7351aa31800245659edc9837e27daffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7351aa31800245659edc9837e27daffd" member_type: VOTER last_known_addr { host: "127.31.69.1" port: 34877 } } peers { permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 } } peers { permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 } }
I20250623 08:04:52.306219 343 tablet_bootstrap.cc:654] T ca5c694856f44201a8aa4314a4f81681 P 125745849172449ba4be8244d61fdcb3: Neither blocks nor log segments found. Creating new log.
I20250623 08:04:52.306107 952 ts_tablet_manager.cc:1939] T c74e8c697d244ac2ac8174bb1fc76e61 P 59f7c614254f42009306b13337f06576: Deleting consensus metadata
I20250623 08:04:52.308198 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 73) took 2719 ms. Trace:
I20250623 08:04:52.313199 32651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba5913811e9343389c9ea0ec8b8c05ef" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "125745849172449ba4be8244d61fdcb3" is_pre_election: true
I20250623 08:04:52.313371 32651 raft_consensus.cc:2466] T ba5913811e9343389c9ea0ec8b8c05ef P 125745849172449ba4be8244d61fdcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:52.315524 32369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba5913811e9343389c9ea0ec8b8c05ef" candidate_uuid: "59f7c614254f42009306b13337f06576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7351aa31800245659edc9837e27daffd" is_pre_election: true
I20250623 08:04:52.315701 32369 raft_consensus.cc:2466] T ba5913811e9343389c9ea0ec8b8c05ef P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59f7c614254f42009306b13337f06576 in term 0.
I20250623 08:04:52.318063 2874 consensus_queue.cc:1035] T 1ae0a411a3e54c45b8d5c5b9b0eea762 P 7351aa31800245659edc9837e27daffd [LEADER]: Connected to new peer: Peer: permanent_uuid: "125745849172449ba4be8244d61fdcb3" member_type: VOTER last_known_addr { host: "127.31.69.3" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250623 08:04:52.309162 952 rpcz_store.cc:276] 0623 08:04:49.588481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.591306 (+ 2825us) service_pool.cc:225] Handling call
0623 08:04:52.238046 (+2646740us) ts_tablet_manager.cc:1145] Deleting tablet c74e8c697d244ac2ac8174bb1fc76e61
0623 08:04:52.238055 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:52.275946 (+ 37891us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:52.302003 (+ 26057us) tablet_metadata.cc:641] Metadata flushed
0623 08:04:52.308179 (+ 6176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8173,"tablet-delete.queue_time_us":2646403}
I20250623 08:04:52.324733 1213 catalog_manager.cc:4928] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): tablet c74e8c697d244ac2ac8174bb1fc76e61 (table default.Table_5785d39307cb42758ded4062307c55e3 [id=4ceb1e1117f747b3a5424c01cd47bd6b]) successfully deleted
W20250623 08:04:52.282912 32589 leader_election.cc:343] T f20b23fd6e5c44f58b1fb2bf617c5704 P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
W20250623 08:04:52.326350 32589 leader_election.cc:343] T 13824916241c4cf596119d44d89cd17f P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:52.325927 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 74) took 2736 ms. Trace:
W20250623 08:04:52.326690 32589 leader_election.cc:343] T ec4d50d5f9394afe97e3b43d65e6894e P 125745849172449ba4be8244d61fdcb3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7351aa31800245659edc9837e27daffd (127.31.69.1:34877): Illegal state: must be running to vote when last-logged opid is not known
I20250623 08:04:52.327564 32364 raft_consensus.cc:3058] T e41b15c89a744a698ca5eee2f143b4bc P 7351aa31800245659edc9837e27daffd [term 0 FOLLOWER]: Advancing to term 1
I20250623 08:04:52.326700 952 rpcz_store.cc:276] 0623 08:04:49.588908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.591490 (+ 2582us) service_pool.cc:225] Handling call
0623 08:04:52.324985 (+2733495us) ts_tablet_manager.cc:1145] Deleting tablet 7ef27bb5b1764ef09888fb0dfcae5be7
0623 08:04:52.324997 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:52.325710 (+ 713us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2733182}
I20250623 08:04:52.327869 2718 consensus_queue.cc:1035] T 3fcd0382328f4b0f9ac510c83eb1f68f P 125745849172449ba4be8244d61fdcb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f7c614254f42009306b13337f06576" member_type: VOTER last_known_addr { host: "127.31.69.2" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250623 08:04:52.328521 1213 catalog_manager.cc:4908] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 7ef27bb5b1764ef09888fb0dfcae5be7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ef27bb5b1764ef09888fb0dfcae5be7 already in progress: creating tablet
I20250623 08:04:52.329416 952 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34076 (request call id 75) took 2740 ms. Trace:
I20250623 08:04:52.333012 952 rpcz_store.cc:276] 0623 08:04:49.589012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0623 08:04:49.591701 (+ 2689us) service_pool.cc:225] Handling call
0623 08:04:52.328824 (+2737123us) ts_tablet_manager.cc:1145] Deleting tablet 92005abc61dd40ef9da8124cadc29b6a
0623 08:04:52.328841 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0623 08:04:52.329398 (+ 557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2736836}
W20250623 08:04:52.334684 1213 catalog_manager.cc:4895] TS 59f7c614254f42009306b13337f06576 (127.31.69.2:40197): delete failed for tablet 92005abc61dd40ef9da8124cadc29b6a because the tablet was not found. No further retry: Not found: Tablet not found: 92005abc61dd40ef9da8124cadc29b6a
I20250623 08:04:52.335233 952 tablet_replica.cc:331] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576: stopping tablet replica
I20250623 08:04:52.336541 952 raft_consensus.cc:2241] T f5f58f55070344ff98a3808f70e0c755 P 59f7c614254f42009306b13337f06576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250623 08:04:52.338325 32648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ae0a411a3e54c45b8d5c5b9b0eea762" candidate_uuid: "